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Preface 


The  purpose  of  this  study  was  to  evaluate  corrective 
action  team  (CAT)  leader  training,  a  group  problem  solving 
and  team  dynamics  course  offered  as  part  of  the  growing 
quality  initiatives  at  Aeronautical  Systems  Division  (ASD). 

I  chose  the  subject  because  I  believe  in  the  idea  of 
providing  the  worker  with  tools  that  improve  their 
effectiveness  both  in  and  out  of  the  work  environment. 

Since  I  was  planning  on  spending  the  next  ten  months  of  my 
life  on  a  research  project,  I  thought  it  important  to  find  a 
relevant  subject  where  I  could  make  an  immediate 
contribution.  Evaluating  CAT  leader  training  met  my 
requirements . 

In  the  course  of  the  research  I  was  involved  in  several 
different  training  classes  and  reviewed  the  responses  of 
over  200  course  graduates.  I  found  each  class  to  be  a 
unique  group  of  professionals  consisting  of  employees  with 
an  honest  desire  to  improve  themselves  and  the  organization. 
An  impressive  fifty-seven  percent  of  the  graduates  surveyed 
took  time  from  their  schedules  to  respond  to  the 
questionnaire  showing  a  true  desire  to  improve  the  training 
provided  by  the  Total  Quality  office.  Many  graduates 
provided  excellent  comments  regarding  the  training. 

This  thesis  was  a  team  effort  and  would  not  have  been 
possible  without  the  help  of  many.  I  would  like  to  thank 
the  ASD  Total  Quality  Office  for  all  their  support,  time, 
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and  effort  both  for  this  research  and  the  culture  change 
they  are  facilitating  at  ASD.  Especially  helpful  were  Dale, 
Betty,  KeeKee,  and  Maybelle. 

I  appreciate  the  guidance  of  Lt  Col  Art  Rastetter,  my 
advisor,  who  believed  in  the  research  and  made  room  for  me 
in  his  busy  schedule  as  a  director  of  logistics  for  a  major 
program  office.  More  than  once,  he  scoured  his  own  library 
to  find  information  helpful  for  this  research  and  provided 
continual  encouragement  and  advice  throughout  the  effort. 
Thanks  also  to  Dr.  Guy  Shane  from  APIT's  Department  of 
Communication  and  Organizational  Sciences.  As  a  true  expert 
on  research,  he  made  numerous  contributions  through  his 
review  of  many  drafts  and  revisions. 

On  a  personal  note,  I  would  like  to  thank  my  future 
wife  Mary,  who  endured  both  the  AFIT  program  and  often  a 
stressed-out  fiance  over  the  past  fifteen  months.  She  was 
always  encouraging  and  has  taken  on  the  majority  of  the 
wedding  plans  to  free  up  my  time  for  school.  I  cannot 
imagine  a  bigger  supporter  or  a  finer  mate. 

Final  thanks  go  to  the  God  who  rules  my  life.  To  him 
who  is  able  to  keep  you  from  falling  and  to  present  you 
before  his  glorious  presence  without  fault  and  with  great 
joy--  to  the  only  God  our  Savior  be  glory,  majesty,  power 
and  authority,  through  Jesus  Christ  our  Lord,  before  all 
ages,  now  and  forevermore!  Amen.  (Jude  24,25) 

Kirk  Streitmater 
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Abstract 

-  A 

This  study  evaluated  corrective  action  team  (CAT) 
leader  training,  a  team  dynamics  and  group  problem  solving 
course  provided  by  an  Air  Force  product  division.  A 
pre/post  knowledge  test,  using  developed  behavioral 
objectives,  was  administered  to  61  participants  as  part  of 
an  internal  evaluation  while  373  past  participants  were 
provided  a  questionnaire  for  a  field  evaluation. 

Results  from  the  internal  evaluation  indicated  that 
students  learned  the  majority  of  training  material  presented 
in  the  course.  Self-efficacy  increased  in  most  cases 
showing  an  improved  sel f -perception  regarding  the  ability  to 
perform  training  related  tasks.  Predictors  for  a  pre-test 
score  model  were  identified,  but  overall  provided  limited 
explanatory  power  to  predict  a  student's  pre-score. 

The  field  evaluation  showed  that  the  majority  of 
graduates,  152  out  of  208  (73%),  do  not  function  as  CAT 
leaders.  Those  who  do  lead  CATs,  are  assigned  an  issue 
within  three  months  of  training  and  consider  the  issues 
worked  relevant  to  a  group  problem  solving  approach.  Half 
of  the  ten  tools  and  techniques  taught  in  training  were  used 
by  the  majority  of  graduates,  while  the  other  half  saw 
little  use.  All  are  perceived  as  having  at  least  some 
degree  of  usefulness  by  the  majority  of  respondents.-^ 
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EVALUATION  OF  CORRECTIVE  ACTION  TEAM  (CAT)  LEADER 
TRAINING  IN  AERONAUTICAL  SYSTEMS  DIVISION 


I.  Introduction 


General  Issue 

Aeronautical  Systems  Division  (ASD)  leadership  has 
embraced  the  concept  of  Total  Quality  Management  (TQM)  for 
the  organization  (Roth,  et  al . ,  1990;iii,2).  The  corrective 
action  team  (CAT)  is  one  initiative  that  encourages  employee 
involvement  in  TQM.  CATs  are  formed  on  an  as  needed  basis 
to  look  into  improvement  opportunities  in  the  work 
environment  (ASD/TQ,  undated: 3).  As  problems  are  identified 
in  an  organization,  management  can  use  u  pool  of  trained  CAT 
leaders  to  facilitate  problem  resolution  by  collecting  and 
working  with  a  team  of  people  involved  in  the  process. 

Training  for  CATs  is  provided  to  the  CAT  leader  and 
involves  a  four  day  off-site  seminar  program  in  group 
problem  solving  and  team  dynamics  (ASD/TQ,  undated:3).  This 
training  program  and  the  associated  resources  for  the  four 
day  seminar  represent  a  serious  commitment  from  management 
to  provide  employees  of  ASD  with  the  tools  to  effectively 
solve  problems  at  their  level. 

Currently,  over  1,600  people  or  15%  of  the  ASD 
population  is  involved  in  some  form  of  quality  related  team 
(Roth,  et  al . , 1990 : 11) .  ASD's  ultimate  goal  is  increasing 
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the  "empowerment,  involvement,  effectiveness,  and 
productivity  of  all  employees."  (Roth,  et  al . ,  1990:12)  CAT 
leader  training  is  recognized  as  one  of  the  methods  to 
accomplish  this. 

To  work  toward  that  end,  this  study  was  proposed  and 
accomplished  to  identify  the  effectiveness  of  the  current 
CAT  leader  course,  and  propose  improvements  for  the  future. 
As  with  any  training  program,  the  key  to  effective 
implementation  is  a  continual  evaluation  of  the  product  to 
ensure  it  is  accomplishing  its  objective  (Gagne  and  Briggs, 
1979:36-38)  . 

Specific  Problem 

Nineteen  ASD  CAT  Leader  training  sessions  have  been 
held  involving  over  440  participants.  To  date,  there  has 
been  no  formal  review  of  the  program.  This  deficiency  was 
noted  in  a  recent  evaluation  of  their  Total  Quality  program 
by  the  Federal  Quality  Institute  when  ASD  competed  for  the 
Quality  Improvement  Prototype  (QIP)  Award.  The  report 
identified  "little  evidence  of  evaluations  done  on 
effectiveness  of  training  programs"  (Franke,  1990).  In 
addition  to  this  report.  Lieutenant  General  Thomas  R. 
Ferguson,  the  Commander  of  ASD,  has  placed  a  strong  emphasis 
on  the  development  of  metrics  both  in  the  area  of  quality 
initiative  results  and  the  organizational  effects  of  quality 
training  and  programs  (Roth  et  al . ,  1990:5). 
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Both  the  need  for  training  evaluation  and  the 
development  of  quality  related  metrics  are  addressed  by  this 
study.  It  provides  a  beginning  for  a  comprehensive 
evaluation  of  the  training  program  that  includes  both  an 
internal  and  field  evaluation.  In  addition,  it  provides  a 
set  of  measurements  that  identify  the  current  status  of  CAT 
training  and  implementation  and  lays  the  groundwork  for 
future  longitudinal  studies  on  those  same  areas.  The 
specific  objectives  that  describe  the  research  accomplished 
are  identified  below. 


Research  Objectives 

This  research  had  six  major  objectives  designed  to 
improve  the  quality  and  effectiveness  of  the  CAT  leader 
training  program.  These  six  objectives  were: 

1.  Identify  and  document  the  specific  material 
provided  as  part  of  the  current  course  curriculum. 

2.  Determine  if  the  CAT  leader  training  course 
facilitates  learning  of  the  course  material  and 
improves  a  trainee's  self-efficacy  regarding 
application  of  course  material 

3.  Determine  the  characteristics  of  current  corrective 
action  teams  and  the  involvement  of  graduates. 

4.  Identify  the  level  of  self-efficacy  of  past 
attendees  regarding  application  of  course  material. 

5.  Identify  how  past  attendees  perceive  and  apply  the 
problem  solving  tools  and  techniques  provided  in  the 
course . 

6.  Based  on  the  above  objectives,  provide  the 
Commander  of  ASD  with  recommendations  for  improvements 
to  the  current  CAT  leader  training  curriculum. 

These  objectives  provide  a  guideline  for  the  research 

and  determine  the  scope  of  the  work  proposed.  To  further 
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define  research  requirements  and  reach  the  above  objectives, 
more  specific  research  questions  are  proposed. 


Investigative  Research  Questions 

The  investigative  research  questions  provided  the 
specific  guidelines  for  the  research  and  represented  a  more 
detailed  breakdown  of  the  research  objectives  identified 
above.  Each  is  individually  addressed  in  the  presentation 
and  discussion  of  results  chapter  of  the  paper.  The 
investigative  questions  include: 

1.  Is  the  basic  curriculvun  described  by  the 
information  in  the  CAT  leader  notebook  taught  by  all 
course  facilitators? 

2.  Does  CAT  leader  training  improve  the  knowledge  of 
participants  in  each  of  the  training  blocks  presented? 

3.  Are  the  following  antecedent  variables  significant 
predictors  of  a  student's  performance  in  the  knowledge 
pre-test  (Y)? 

XI.  Involvement  in  previous  CATs  (Experience) 

X2.  Previous  problem  solving/team  dynamics 
training  (Training) 

X3.  Education  Level  (Education) 

4.  Is  there  an  overall  increase  in  self-efficacy  in 
the  application  of  course  material  as  a  result  of 
attending  CAT  leader  training? 

5.  Are  a  significant  number  of  CAT  leaders  assigned 
CAT  issues  within  three  month  of  training? 

6.  Are  there  a  significant  number  of  training 
participants  becoming  members  of  CATs? 

7.  Is  there  a  noticeable  decline  in  self-efficacy 
regarding  application  of  training  material  as  time 
since  training  increases? 

8.  Are  problem  solving  tools  and  techniques  being  used 
in  the  organizational  and  CAT  environments? 
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9.  Are  problem  solving  tools  and  techniques  perceived 
as  being  used  frequently  in  the  CAT  and  organizational 
environment? 

10.  Are  problem  solving  tools  and  techniques  perceived 
as  being  useful  tools  for  team  problem  solving? 

11.  Do  a  significant  number  of  CAT  leaders  rate  the 
use  of  a  CAT  as  appropriate  in  addressing  the  CAT- 
related  issues  assigned? 

Scope 

This  study  was  limited  to  evaluating  CAT  leader 
training  in  Aeronautical  Systems  Division.  Conclusions 
drawn  cannot  be  applied  to  other  organizations.  Review  of 
the  current  training  program  was  specifically  emphasized  in 
the  research  with  sources  of  information  that  included  the 
CAT  leader  notebook  (ASD/TQ,  undated),  the  course  syllabus, 
past  course  evaluation  forms,  and  discussions  with 
instructors . 

Research  consisted  of  three  parts:  development  of 
objectives,  a  pre/post  knowledge  test  administered  during 
the  course  and  a  field  questionnaire  for  past  participants. 
The  purpose  of  developing  course  objectives  was  primarily  to 
formalize  the  current  desired  behavioral  outcomes  of  the 
training  rather  than  to  evaluate  the  syllabus  currently  in 
place.  This  resulted  in  observations  regarding  material  in 
the  course  curriculum,  but  did  not  involve  discussing  what 
should  be  in  a  course  for  CAT  leaders. 

Knowledge  testing  accomplished  during  the  training  as 
part  of  this  project  focused  primarily  on  the  knowledge  and 
comprehension  levels  of  learning  in  the  cognitive  domain 
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(Bloom,  et  al . ,  1971).  Research  into  higher  levels  of 
learning  such  as  analysis,  synthesis  and  evaluation  is 
reserved  for  follow-on  projects. 

The  field  questionnaire  solicited  feedback  on  the  use 
of  CAT  leader  course  material  and  the  current  conditions  and 
procedures  of  CATs  in  ASD.  Of  major  interest  was  the 
applicability  and  use  of  the  problem  solving  tools  and 
techniques  currently  part  of  the  course  curriculum.  The 
majority  of  the  participants  were  from  Aeronautical  Systems 
Division  with  other  participants  representing  Air  Force 
Systems  Command,  Air  Force  Logistics  Command,  and  Defense 
Electronic  Systems  Center.  While  over  400  individuals  have 
received  CAT  leader  training,  the  field  research  only  dealt 
with  those  373  Air  Force  personnel  who  report  to  the 
commander  of  ASD,  whose  organization  sponsors  the  training. 
This  simplified  the  research,  since  coordination  with  other 
Air  Force  organizations  was  not  required  to  gather  data. 

Summary 

This  research  was  only  the  first  step  in  evaluating  CAT 
leader  training  and  hopefully  provides  a  basis  for  more  in- 
depth  study  to  improve  the  course  curriculum  and  ultimately 
its  effect  on  the  organization. 

The  following  chapter  provides  background  to  TQ  and  CAT 
leader  training  in  Aeronautical  Systems  Division.  Chapter  3 
provides  a  literature  review  associated  with  learning 
theories  and  training  program  evaluation.  Chapter  4 


6 


describes  the  research  methodology  used.  Chapter  5  then 
discusses  the  results  of  the  evaluation  that  lead  to 
conclusions  and  recommendations  provided  in  Chapter  6. 
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II.  Background 


Introduction 

This  chapter  provides  the  background  of  CAT  leader 
training.  While  the  training  is  applicable  to  many 
organizational  situations,  its  primary  emphasis  has  been  for 
use  in  the  organizational  total  quality  structure.  A  brief 
discussion  of  that  structure  follows,  along  with  information 
on  the  development,  format,  and  content  of  the  course 
itsel f . 

Total  Quality  in  Aeronautical  Systems  Division 

Aeronautical  Systems  Division  (ASD)  leadership  has 

embraced  the  concept  of  Total  Quality  Management  (TQM)  for 

the  organization  (Roth,  et  al . ,  1990:iii,2).  The  ASD 

definition  of  Total  Quality  Management  was  identified  in  an 

executive  off-site  in  1988  as: 

A  leadership  philosophy  that  creates  a  working 
environment  which  promotes  trust,  teamwork,  and 
the  quest  for  continual  improvement.  (Roth,  et 
al.,  1990:1) 

As  a  result  of  this  definition  and  the  vision  that  ASD 
would  strive  to  remain  "the  center  of  excellence  for 
research,  development  and  acquisition  of  aerospace  systems," 
(Roth,  1990:1)  ASD  developed  a  set  of  TQ  principles  to 
describe  how  it  will  operate  to  encourage  a  TQ  environment 
(Roth,  et  al . ,  1990:2).  These  principles  include: 

1.  Know  and  satisfy  the  customers'  needs. 

2.  Delegate  responsibility  and  authority  -  accept 
accountabi 1 ity . 
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3.  Give  EVERYONE  a  stake  in  the  outcome. 

4.  Set  goals,  compete,  measure  progress,  and  reward. 

5.  Create  a  climate  of  pride,  professionalism, 
excellence,  and  trust. 

6.  Strive  for  continuous  improvement  -  Make  It 
Better!  (Ridgely,  undated: 6) 

To  facilitate  incorporating  these  principles  into  the 
ASD  organizational  culture  a  TQ  structure  was  created  in 
each  reporting  organization.  Over  the  next  two  years,  ASD 
senior  leadership  focused  their  efforts  on  several  key 
issues  to  improve  the  quality  culture.  These  included 
emphasis  on  the  following  areas:  quality  environment, 
quality  measurement,  quality  improvement  planning,  employee 
involvement,  employee  training  and  recognition,  quality 
assurance,  and  customer  focus  (Roth,  et  al . ,  1990:1-17). 
Progress  in  the  Total  Quality  program  in  ASD  prompted 
management  to  submit  an  application  and  be  selected  as  one 
of  ten  finalists  for  the  1991  Quality  Improvement  Prototype 
(QIP)  Award  (Franke,  1990).  The  structure  for  the  TQ 
organization  established  is  described  below. 

Organizational  TQ  Structure 

Nithin  each  reporting  organization  in  ASD,  a  TQ 
structure  has  been  developed  to  facilitate  the  incorporation 
of  the  quality  principles  discussed  above  into  the 
organizational  culture  (Ridgely,  undated:12).  These 
organizations  mainly  consist  of  system  program  offices 
(SPOs)  that  manage  a  particular  weapon  system,  functional 
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deputies  that  provide  matrix  support  to  SPOs,  research  and 
development  laboratories  and  a  flight  test  wing.  Each 
organization  has  the  same  general  total  quality  management 
structure  as  shown  in  Figure  1  and  defined  below: 


Figure  1.  Generic  TQ  Structure  in  ASD  (Roth,  et  al . , 
1990:3;  Ridgely,  undated:12) 


The  Executive  Steering  Committee  (ESC).  The  Executive 
Steering  Committee  (ESC)  consists  primarily  of  the 
organization's  main  leaders  and  develops  the  policy  and 
strategic  plans  regarding  TQ.  It  establishes  the 
organization's  vision,  goals,  and  objectives  and  oversees  TQ 
by  providing  resources  and  demonstrating  management 
commitment  (Roth,  et  al . ,  1990:4). 
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The  TO  Team.  The  TQ  Team  is  seen  as  the  primary 
instrvunent  in  the  TQ  structure.  It  is  made  up  of  a  cross 
section  of  organizational  personnel  and  is  responsible  for 
designing,  implementing  and  managing  a  system  to  facilitate 
solving  organizational  problems  (Ridgely,  undated:12).  To 
accomplish  this,  four  subsystems  are  normally  implemented. 
These  include  the  Education  subcommittee  to  energize  and 
educate  the  organization  on  Total  Quality,  the  Search  for 
Opportunities  (SFO)  subcommittee  that  focuses  on  generating 
employee-related  ideas,  the  Corrective  Action  subcommittee 
that  uses  corrective  action  teams  to  solve  problems 
identified  by  employees,  and  the  Measurement  subcommittee 
that  measures  progress  in  solutions  and  key  objectives 
(Roth,  et  al . ,  1990:4) . 

The  Corrective  Action  Team  (CAT).  In  problem 
resolution,  the  corrective  action  team  (CAT)  is  commissioned 
by  the  Corrective  Action  subcommittee  or  Executive  Steering 
Committee  to  work  specific  problems.  The  team  is  comprised 
of  individuals  inside  the  organization  who  are  knowledgeable 
about  aspects  related  to  the  problem.  The  CAT  is  led  by  a 
CAT  leader  who  has  received  specific  training  in  group 
problem  solving  techniques  and  team  dynamics  via  CAT  leader 
training.  Other  team  members  receive  no  training,  but  look 
towards  the  CAT  leader  to  teach  the  methodology  needed  to 
facilitate  team  effectiveness.  The  CAT  is  formed  to  address 
a  specific  problem,  and  after  problem  resolution,  is 
disbanded.  Once  the  CAT  is  dissolved,  the  CAT  leader  is 
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then  available  for  reassignment  to  another  issue  (ASD/TQ, 
undated: 3) . 

The  Critical  Process  Team  (CPTl.  The  final  team 
involved  in  the  TQ  structure  is  the  critical  process  team 
(CPT),  which  is  chartered  by  the  ESC  to  investigate  "high 
level,  cross-organizational  processes  which  have  a  critical 
impact  on  satisfying  the  customer's  requirement."  (Roth,  et 
al . ,  1990:4)  These  teams  are  similar  to  CATs,  except  the 
issues  addressed  are  more  complex  and  the  entire  team 
receives  problem  solving  training  together  (Ridgely, 
undated : 14) . 

Search  For  Opportunity  (SFO).  The  Search  For 

Opportunity  (SFO)  is  a  method  of  encouraging  participation 

by  soliciting  employee  ideas  and  improvements  through  the 

formal  TQ  structure.  The  employee  inputs  are  normally 

submitted  on  a  form  designed  specifically  by  the 

organization  and  placed  in  accessible  locations  throughout 

the  work  area.  A  completed  form  receives  prompt  attention 

and  is  formally  tracked  to  ensure  a  submitter’s  concerns, 

ideas  or  suggestions  receive  adequate  consideration.  SFOs 

fall  into  one  of  three  types.  These  "types"  of  SFOs  aid  in 

tracking  and  assigning  issues  to  the  appropriate  owners  and 

organizations.  The  three  types  of  SFOs  include: 

Type  1:  The  issue  falls  within  the  control  of  a 
single  office  that  has  the  power  to  make  changes,  if 
appropriate. 

Type  2:  The  issue  may  involve  coordination  by 
several  offices  in  an  organization  and  normally  is  a 
prime  candidate  for  a  corrective  action  team  (CAT). 
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Type  3:  The  issue  is  beyond  the  scope  of 
influence  for  an  organization  and  is  sent  up  to  the  ASD 
level  (or  higher)  to  be  addressed.  These  SFOs  are 
formally  tracked  by  ASD/TQ.  (ASD/TQ,  undated:10) 


The  SFO  is  the  basis  of  employee  involvement  in  the  TQ 
culture  and  is  the  first  form  of  documentation  that  defines 
an  issue.  They  often  provide  the  rationale  used  to 
establish  a  CAT  or  CPT.  Unlike  a  suggestion  system, 
solutions  are  not  a  requirement  when  the  SFO  is  submitted 
(Roth,  et  al ,  1990) . 

Now  that  the  TQ  structure  has  been  discussed,  further 
detail  on  the  specifics  of  CAT  leader  training  are  provided. 

CAT  Leader  Training 

Currently,  training  is  accomplished  monthly  in  a  4-day 
off-site  seminar  format  that  begins  Tuesday  morning  and  ends 
Friday  afternoon.  The  overall  objective  involves  developing 
CAT  leaders  who  can  successfully  "lead  a  CAT  through  the 
problem  solving  process  to  implementation  of  a  solution  to 
an  organizational  issue."  (ASD/TQ,  undated:Tab)  Specific 
course  objectives  currently  include  training  a  CAT  leader: 

1.  To  be  able  to  organize  a  corrective  action  team  and 
manage  the  group  dynamics  so  that  the  CAT  develops  into 
and  remains  and  effective,  functioning  team. 

2.  To  be  able  to  explain  the  TQ  structure  and 
philosophy  to  individual  members  and  how  the  CAT  fits 
into  the  overall  TQ  process. 

3.  To  be  able  to  teach  the  problem  solving  methodology 
and  the  appropriate  tools  and  techniques  to  CAT  members 
so  that  they  are  able  to  effectively  use  them  in  the 
problem  solving  process.  (ASD/TQ,  undated:Tab) 
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As  noted  by  the  above  three  objectives,  CAT  leader 
training  focuses  on  three  key  issues:  group  or  team 
dynamics,  the  TQ  structure  and  philosophy,  and  the  problem 
solving  process. 

Background  and  Transition  to  Internal  Training.  The 
majority  of  ASD's  transition  to  a  TQ-focused  culture  was 
facilitated  by  the  Cumberland  Group,  a  former  subsidiary  of 
AARMCO  Steel  that  formed  their  own  TQ  consulting  company. 
Initially  Cumberland  accomplished  all  TQ-related  training 
for  ASD  through  a  support  contract  (ASD/TQ,  undated:!). 

Early  on,  a  plan  to  transition  the  education  and  training 
responsibility  from  Cumberland  to  ASD  was  developed  to 
establish  an  eventual  ASD  sel f '■contained  training  program 
that  encouraged  a  TQM  culture  working  for  constant 
improvement . 

CAT  leader  training  was  the  first  training  program 
scheduled  to  be  completely  supported  by  ASD  personnel,  which 
included  a  group  of  specially  trained  facilitators  for  the 
program. 

Course  Facilitators.  With  the  plan  to  transition  the 
CAT  leader  training  responsibilities  in  1989  from  Cumberland 
to  ASD,  an  organization-wide  request  for  facilitator 
volunteers  was  conducted.  Those  individuals  interested  in 
becoming  a  facilitator  required  the  approval  of  their 
management  to  submit  an  application.  Twelve  part-time  CAT 
leader  course  facilitators  were  chosen  in  ASD  and  in  July  of 
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1989  attended  a  1  week  off-site  dry  riin  for  the  pilot 
program  (Ball,  1991). 

On  September  1  of  1989,  the  first  ASD  sponsored  CAT 
leader  training  was  conducted.  The  exclusive  use  of  part- 
time  facilitators  was  continued  until  April  of  1990  when 
full-time  coordinators  were  added  to  the  ASD  Commander's 
Office  for  Total  Quality  (ASD/TQ)  staff  (Ball,  1991).  In 
addition  to  working  with  part-time  facilitators  on  CAT 
leader  training,  these  full-time  facilitators  also  conducted 
other  TQ  related  training,  served  as  a  facilitator  resource 
for  individual  ASD  organizations,  and  supported  various 
management  functions  on  quality  at  the  ASD  level. 

As  of  February  1991,  five  of  the  12  original  part-time 
facilitators  were  still  involved  in  CAT  leader  training. 

Two  have  become  full-time  facilitators  for  ASD  and  continue 
with  their  involvement  in  the  training. 

New  facilitators  are  introduced  to  the  training  by 
attending  the  CAT  leader  course  as  a  student  and  then  at 
least  once  as  an  observer.  Once  the  new  facilitator  feels 
thoroughly  comfortable  with  the  material  and  the 
presentations,  he/she  may  start  leading  course  discussions. 
The  materials  provided  to  facilitators  for  class  preparation 
include:  an  instructor's  notebook  (developed  by  the 

Cumberland  Group),  a  CAT  leader  course  notebook  provided  to 
all  participants,  and  various  material  associated  with 
training  exercises.  Additional  support  for  new  personnel  is 
provided  by  current  course  instructors  and  a  Cumberland 
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Group  facilitator  training  course  that  is  included  in  its 
current  support  contract  with  ASD  (Wissman,  1991). 

Course  Administration.  A  CAT  leader  training  course  is 
normally  scheduled  several  months  in  advance  by  the  ASD 
Commander's  Office  for  Total  Quality  and  follows  the  format 
provided  by  the  syllabus  in  Appendix  A.  Each  course 
averages  approximately  19  attendees  and  is  facilitated  by 
two  to  three  instructors  with  course  experience.  Normally 
two  facilitators  are  assigned  to  each  course,  but  others  may 
be  added  to  account  for  previous  scheduling  conflicts,  new 
instructors,  or  group  size. 

The  facilitator  team  meets  prior  to  the  course  and 
agrees  on  team  norms  which  define  roles,  responsibilities 
and  procedures  for  the  course.  Team  norms  often  include 
allocation  of  facilitator  duties  and  briefing 
responsibilities,  selection  of  specific  problem  exercise 
examples  to  reinforce  course  topics  and  determining  meeting 
times  to  discuss  possible  schedule  changes. 

A  syllabus  defines  the  course  material,  but  facilitator 
teams  have  a  large  amount  of  flexibility  to  tailor  the 
training  material  for  each  course  offering.  Several  team 
exercises  are  available  for  use  during  the  training  to 
emphasize  important  topics  and  provide  hands-on 
reinforcement  of  training  concepts. 

Course  Content.  The  course  runs  for  four  consecutive 
days  and  covers  six  different  training  blocks  that  are 
identified  in  Figure  2.  They  include  an  overview  of  TQM, 
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effects  of  personal  values,  group  dynamics,  the  problem 
solving  process,  problem  solving  tools  and  techniques  and 
meeting  effectiveness.  This  curriculum  is  designed  to 
accomplish  the  3  terminal  objectives  identified  at  the 
beginning  of  this  section. 


1.  Overview  of  Total  Quality  Management  (TQM) 

2.  Effects  of  Personal  Values 

3.  Group  Dynamics 

4.  The  Problem  Solving  Process 

5.  Problem  Solving  Tools  and  Techniques 

6.  Meeting  Effectiveness _ 

Figure  2.  CAT  Leader  Training  Blocks  (ASD/TQ,  undated) 

Exercises  and  videos  are  also  an  important  part  of  the 
curriculum  and  are  used  after  many  presentations  to 
reinforce  the  topics  discussed.  Often  the  exercises  involve 
small  teams  of  participants  getting  together  to  practice 
techniques  by  working  a  realistic  problem  scenario. 

The  last  day  of  the  course  involves  a  large  exercise 
where  three  to  four  teams  are  provided  a  problem  that 
involves  reduced  attendance  to  the  base  cafeteria.  Each 
group  is  encouraged  to  use  the  material  learned  during  the 
week  to  make  recommendations  to  management.  Facilitators 
take  on  several  roles  during  the  exercise  to  provide 
information  to  the  teams  and  then  represent  management 
during  the  out  brief.  Feedback  is  provided  to  all  groups 
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with  the  use  of  material  being  emphasized  rather  than  the 
final  conclusions. 

Current  Status.  CAT  leader  training  has  now  been 
taught  by  an  ASD  cadre  to  over  440  personnel  in  23  different 
course  offerings.  As  of  1990,  more  than  160  quality  related 
teams  including  CATs,  CPTs,  and  TQ  Teams  have  been 
functioning  in  ASD  (Roth,  et  al . ,  1990:11).  To  date,  there 
has  been  no  formal  evaluation  of  the  course  to  assess  its 
effectiveness  in  ASD.  As  noted  earlier,  ASD  was  selected  as 
one  of  ten  finalists  for  the  1991  Quality  Improvement 
Prototype  (QIP)  Award.  One  area  of  needed  improvement 
noted  by  the  Federal  Quality  Institute,  who  judged  the  QIP 
competition,  regarded  the  evaluation  of  training 
effectiveness.  As  a  result  of  this  finding,  and  Lieutenant 
General  Ferguson's  desire  to  identify  metrics  for  evaluating 
TQ  effectiveness  in  ASD,  the  Commander's  Office  for  Total 
Quality  agreed  to  sponsor  this  research.  This  research, 
along  with  the  course  evaluations  that  have  been  utilized 
throughout  the  training  program  should  provide  an  avenue  for 
assessing  training  effectiveness  and  improving  the  current 
course . 

Summary 

ASD  Total  Quality  structure  and  CAT  leader  training 
specifics  were  discussed  to  familiarize  the  reader  with  the 
research  topic.  Discussion  of  the  current  status  of  the  CAT 
leader  course  identified  the  level  of  training  accomplished 
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to  date  and  provided  a  background  and  rationale  for  this 


effort . 
learning 


Chapter  3  follows  with  a  background  discussion  on 
and  evaluating  training  programs. 
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III.  Literature  Review 


Introduction 

The  literature  on  learning,  behavior,  instructional 
theory  and  training  evaluation  is  extensive  and  cannot 
possibly  be  covered  in  this  brief  review.  However,  relevant 
highlights  to  key  theories  will  provide  a  background  and 
summary  of  important  concepts  of  the  research  that  dictated 
the  approach  taken  in  the  effort. 

Theories  of  learning  and  how  learning  is  evaluated  and 
applied  will  provide  a  basis  for  the  discussion  of  the 
instructional  system  design  and  the  systems  approach  to 
training.  Training  evaluation  will  be  specifically 
emphasized  because  of  the  research  focus  as  will  the  use  of 
research  design  in  course  evaluation. 

General  Theories  of  Learning 

The  desired  learning  outcome  is  often  to  invoke  a 
relatively  permanent  change  in  behavior  that  results  from 
experience  (Daft  and  Steers,  1986:48).  While  the  behavior 
change  is  a  central  issue,  both  the  behavioral  and  cognitive 
aspects  of  learning  provide  for  a  more  complete  definition 
and  theory  foundation  (AFM  50:62,  1984:2-1;  Luthans, 
1979:100-101).  Luthans  identifies  an  expanded  concept  of 
learning  as  follows: 

1.  Learning  involves  a  change,  though  not  necessarily 
an  improvement,  in  behavior. 

2.  The  change  in  behavior  must  be  relatively 
permanent  in  order  to  be  considered  learning. 
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3.  Some  form  of  practice  or  experience  is  necessary 
for  learning  to  occur. 

4.  Finally  it  should  be  stressed  that  the  practice  or 
experience  must  be  reinforced  in  some  way  in  order  for 
learning  to  occur.  (Luthans,  1977:281) 

Daft  and  Steers  provide  a  brief  review  of  the  three 

theories  identified  as  fundamental  to  learning  and  related 

to  the  four  points  identified  above.  These  are:  1) 

classical  conditioning;  2)  operant  conditioning;  and  3) 

cognitive  learning  theory.  Both  classical  and  operant 

conditioning  focus  on  a  stimulus  response  condition  in  the 

learning  process  (Daft  and  Steers,  1987:49)  which  places  a 

significant  emphasis  on  the  manipulation  of  the  environment 

to  promote  learning.  Cognitive  theory  focuses  not  on 

environment,  but  on  how  a  learner  "remembers  and  retrieves 

information  from  memory."  (Richey,  1986:65)  The  three 

theories  are  summarized  below. 

Classical  Conditioning  Theory.  Classical  conditioning 

theory  originates  from  the  Russian  physiologist  Pavlov's 

experiments  with  dogs.  In  Pavlov's  famous  experiments,  dogs 

were  taught  to  salivate  at  the  sound  of  a  bell.  In  this 

case,  a  link  between  a  conditioned  stimulus  (a  bell)  and  an 

unconditioned  stimulus  (meat)  was  reinforced  by  introducing 

both  stimuli  simultaneously.  Since  a  dog's  natural  response 

is  to  salivate  at  the  sight  of  meat  it  represented  no 

learned  behavior;  however,  repeated  linking  of  the  bell  and 

meat  resulted  in  the  dog  salivating  at  the  sound  of  the  bell 

even  when  the  meat  was  absent.  In  this  case,  Pavlov 
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concluded  that  learning  had  occurred  because  the  dog  now 
associated  two  normally  unrelated  objects  (Daft  and  Steers, 
1986:50). 

Operant  Conditioning  Theory.  Skinner  in  his  article 
Operant  Behavior  (1963)  introduces  the  concept  of  rewards 
into  the  realm  of  learning  by  noting  that  an  organism  is 
behaving  a  given  way  because  it  expects  a  given  effect 
(503).  His  position  is  an  extension  of  Thorndike's  "law  of 
effect”  which  introduced  the  principle  that  the  effect  of 
action  itself  causes  future  action  to  occur  (Skinner, 
1963:503).  This  "law"  argues  that  behavior  resulting  in 
positive  reinforcement  will  likely  be  repeated,  and  behavior 
with  negative  consequences  will  be  avoided  (Daft  and  Steers, 
1986:51).  In  effect,  learning  occurs  as  a  consequence  of 
behavior  (Luthans,  1977:284).  Daft  and  Steers  identify 
three  important  concepts  of  this  model  that  must  be  present 
for  it  to  apply: 

1.  Drive:  A  felt  need  exists  and  must  be  present 
before  learning  can  take  place. 

2.  Habit:  The  ability  to  develop  an  experienced  bond 
or  connection  between  stimulus  and  response. 

3.  Heinf orcement  or  reward:  The  feedback  or 
consequence  individuals  receive  as  a  result  of  action. 

(Daft  and  Steers,  1986:51) 

The  drive  and  habit  concepts  are  both  considered  part 
of  the  stimulus-response  bond  and  the  stronger  each  is,  the 
stronger  the  motivation  of  the  individual  to  respond  in  a 
certain  way.  Both  classical  and  operant  conditioning  are 
considered  behaviorist  theories  and  are  important  in  that 
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manipulating  the  environment  is  a  key  aspect  of  promoting 
learning  (Fogg,  1990:19). 

Cognitive  Theory.  Cognitive  learning  theorists  contend 
that  much  learning  does  not  involve  the  stimulus-response 
bond  but  is  the  result  of  acquisition,  analysis  and  insight 
of  known  facts  (that  can  include  abstract  concepts  and 
generalizations)  that  allow  more  sophisticated  learning  to 
occur  in  the  proper  environment  (Daft  and  Steers,  1986:52). 
Concepts  such  as  expectancy,  demand,  and  incentive  are 
utilized  as  the  cognitive  model  emphasizes  the  positive  and 
free  will  aspects  of  humans  (Luthans,  1977:100). 

Luthan's  statement  itself  identifies  the  contrast  in 
the  theories.  While  much  of  the  research  that  deals  with 
behavioral  principles  involves  some  use  of  animal  research, 
cognitive  theory  has  focused  primarily  on  human  subjects 
(Richey,  1986:65). 

While  the  initial  acceptance  of  cognitive  theory  was 
limited,  it  has  recently  received  more  attention.  Richey 
(1986)  identifies  one  major  example  by  noting  the  increased 
attention  given  to  instructional  design.  She  also 
references  the  work  of  Gagne  and  White  (1978)  in  identifying 
the  role  of  cognitive  theory  as  a  key  element  of  instruction 
leading  to  learning  and  a  change  in  performance  (66). 

Evaluation  of  Learning 

Although  learning  is  not  a  directly  observable 
phenomena  (Daft  and  Steers,  1986:49),  criterion  standards 


23 


that  allow  you  to  measure  the  amount  of  learning  that  has 
occurred  are  necessary.  The  Taxonomy  of  Educational 
Objectives  is  a  classification  system  of  educational 
objectives  that  was  developed  to  facilitate  communication 
among  researchers  and  others  regarding  objectives,  test 
items,  and  test  procedures  (Bloom,  et  al . ,  1971:39). 

The  three  specific  taxonomies  include  cognitive, 
affective  and  psychomotor.  Of  specific  importance  in  this 
research  are  the  taxonomies  regarding  the  cognitive  and 
affective  domains  identified  in  Figures  3  and  4. 


Cognitive  Domain 

Level  of  Learning 

Mental  Activity 

Evaluation 

Exercise  of  Learned  Judgement 

Synthesis 

Create  new  relationships 

Analysis 

Determine  relationships 

Application 

Use  of  generalizations 

Comprehension 

Translate,  interpret,  extrapolate 

Figure  3.  Bloom's  Taxonomy  of  Educational  Objectives 
(AFM  50-62,  1984:2) 


The  cognitive  domain  taxonomy  was  initially  proposed  by 
Bloom  (1956)  and  involves  intellectual  responses  of  the 
learner  and  the  level  of  mental  activity  regarding  the 
learned  material.  The  affective  domain  (Krathwohl,  et  al . , 
1964)  involves  more  the  internalization  or  level  of 
acceptance  of  the  learned  material  (Bloom  et  al . ,  1979:39; 
Knirk  and  Gustafson,  1986:80-81).  The  taxonomy  levels 


24 


provide  a  hierarchical  framework  to  the  process  where 
behaviors  progressively  get  more  complex,  abstract  and 
internalized  (Bloom,  et  al . ,  1971:39). 


Affective  Domain 

Level  of  Learning 

State  of  Mind 

Characterization 

Incorporates  values  into  life 
style 

Organization 

Rearrangement  of  value  sys  v. , 

Valuing 

Acceptance 

Responding 

Reacts  voluntarily  or  complies 

Receiving 

Willingness  to  pay  attention 

Figure  4.  Krathwohl's  Taxonomy  of  Educational 
Objectives  (AFM  50-62,  1984:2) 

The  use  of  these  taxonomies  allows  a  more  universal 
means  of  measuring  behavioral  outcomes  and  determine  the 
appropriate  level  of  instruction  for  students  (Knirk  and 
Gustafson,  1986:82).  This  fact  seems  critical  in 
considering  the  level  and  type  of  instruction  students 
should  receive  and  how  to  measure  the  impacts  of  that 
instruction. 

Ultimately,  the  taxonomies  provide  a  useful  tool  both 
to  translate  broadly  stated  objectives  into  operational 
terms  and  build  items  to  measure  "higher  order  mental 
processes  or  affective  outcomes  of  instruction."  (Bloom,  et 
al.,  1971:40) 
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Application  of  Learning 


Even  though  the  cognitive  theory  focuses  on  the 
internal  processing  that  results  in  learning,  both  recognise 
the  effects  of  learning  on  the  environment.  As  a  matter  of 
fact,  Skinner  poses  that  the  reason  we  are  interested  in 
behavior  at  all  is  because  of  those  effects  on  the 
environment  (Skinner,  1963:503).  So,  it  is  not  exclusively 
the  environment  in  which  the  learning  occurs  that  matters, 
it  is  the  environmental  impact  of  the  learned  behavior. 
Thorndike's  discussion  of  transferability  from  Crooks 
assessment  of  classroom  evaluation  practices  (1988)  states 
the  point  wel 1 : 

The  crucial  indicator  of  a  student's  understanding  of  a 
concept,  a  principle,  or  a  procedure  is  that  he  is  able 
to  apply  it  in  circumstances  that  are  different  from 
those  under  which  it  was  taught.  Transferability  is  the 
key  feature  of  meaningful  learning.  So  if  we  are  to 
test  for  understanding,  we  must  test  in  circumstances 
which  are  at  least  part  new.  (Crooks,  1988:442-443) 

Crooks  points  out  above  that  a  student's  learning  needs 

to  be  tied  with  the  ability  to  transfer  what  is  taught  into 

a  new  environment.  Several  factors  are  important  to  the 

amount  of  transference  that  occurs  and  relate  to  the  amount 

of  learning  made  by  the  student.  Three  such  factors  are 

retention,  self-efficacy,  and  attitudes. 

Retention.  Retention  is  seen  as  a  principal  effect  of 

learning  and  is  an  important  determinant  of  the  amount  of 

transfer  that  will  eventually  occur  in  a  job  situation 

(Goldstein,  1986:80).  Goldstein  identifies  several  effects 
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on  the  retention  process  that  can  effect  transference  in  the 
area  of  personnel  training: 

1.  The  degree  of  original  learning.  Any  effect  that 
limits  original  learning  will  affect  retention.  These 
include  whole  vs  part  learning,  massed  vs  spaced 
practice,  overlearning,  and  knowledge  of  results. 

2.  Neaningfulness  of  the  material.  The  more 
meaningful  the  material,  the  more  it  is  retained 

3.  The  amount  of  interference.  Both  material  learned 
before  (proactive)  and  after  (retroactive)  can 
interfere  with  the  recall  of  original  material. 

4.  Motives,  perceptions,  and  retention.  Memory  is 
often  affected  by  our  perceptions  of  events  and  the 
state  of  mind  during  the  original  occurrence. 

(Goldstein,  1984:87-88) 

Self-Efficacy .  Bandura  identifies  self  efficacy  as  a 
students'  perception  of  his/her  capability  to  perform 
courses  of  action  with  respect  to  a  certain  situation 
(Bandura,  1982:122).  Crooks  (1988)  notes  that  perceptions 
of  self-efficacy  in  an  area  have  a  high  correlation  with 
ultimate  success  in  that  area.  Even  with  difficult  tasks  or 
encounters  with  failure,  a  higher  self-efficacy  often 
results  in  a  stronger  effort  and  a  higher  level  of  task 
success  (Crooks,  1988:462). 

Attitudes .  The  two  classes  of  attitudes  that  relate  to 
learning  involve  attitudes  that  relate  to  how  readily  an 
individual  learns  and  attitudes  that  are  a  established  or 
changed  as  a  result  of  learning  (Gagne  and  Briggs,  1979:84- 
85).  Since  the  philosophy  of  TQM  involves  a  change  in  the 
organizational  culture  (Roth,  et  al.,  1990:1),  attitudes 
play  an  important  part  in  TQM  success.  This  is  highlighted 
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by  Gagne  and  Briggs'  (1979)  identification  that  values 
function  to  affect  "approaching"  or  "avoiding"  specific 
behaviors  (85).  One  important  result  of  this  condition  is 
that  specific  behavioral  oatcomes  result  from  changes  in 
attitudes . 


The  Learning  Theory  and  Training  Gao 

The  application  of  learning  theory  developed  over  the 
past  decades  would  seem  to  support  an  efficient  and 
effective  transition  to  the  modern  training  and 
instructional  environment,  but  Goldstein  (1984)  along  with 
others  have  found  that  the  significant  progress  made  in 
learning  theory  has  focused  in  the  laboratory  environment 
making  application  difficult  and  slow  (Goldstein,  1984:64). 
Three  reasons  specifically  identified  include: 

1.  The  learning  theorists  have  tended  to  focus  on 
highly  speci..ic  laboratory  experimentation,  which  has 
made  generalizations  to  field  settings  extremely 
difficult . 

2.  Until  recently,  the  learning  theorist  has  ignored 
the  complex  areas  of  human  behavior.  Thus  there  is 
relatively  little  information  available  on  problem 
solving,  perceptual  motor  learning,  concept  learning, 
and  other  topics  directly  relevant  to  the  needs  of  the 
training  specialist. 

3.  The  training  specialist  often  demands  quick  answers 
and  ready  solutions  to  complicated  problems  (McGehee 
and  Thayer,  1961).  When  easy  solutions  are  not 
immediately  apparent,  the  practitioner  often  assumes 
that  the  learning  theorist's  entire  program  is 
irrelevant...  [and] ... ignores  learning  theory  and 
contributes  little  information  of  his  own.  (Goldstein, 
1984:64) 
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To  deal  with  the  transition  from  learning  theory  to 
training  application  the  Instructional  System  Design  was 
developed  and  is  described  below. 


Instructional  System  Design  Theory 

In  the  realm  of  an  instructional  setting,  objectives 
that  define  the  level  of  learning  desired  are  required  to 
establish  methods  of  measuring  that  learning.  In  discussing 
the  learning  environment,  Goldstein  (1984)  referenced 
Gagne's  paper  Military  Training  and  Principles  of  Learning 
(1962)  noting  that  developed  learning  principles  were 
"strikingly  inadequate  to  handle  the  job  of  designing 
effective  training  situations"  (p.  85).  From  this 
deficiency  developed  Gagne's  instructional  design  theory. 

Instructional  design  attempts  to  use  knowledge  from 
learning  research  and  theory  to  develop  an  instructional 
system  that  "will  optimize  learning,  retention,  and 
transfer"  (Goldstein,  1984:99).  Below  are  summaries  of  the 
five  categories  of  learning  outcomes  that  form  a  basis  for 
the  objectives  of  instruction: 

1.  Intellectual  Skills.  The  capabilities  for  making  a 
human  individual  competent.  It  involves  the  knowledge 
of  how  to  do  something  and  draws  on  concepts,  rules, 
and  procedures.  It  is  sometimes  referred  to  as 
procedural  knowledge. 

2.  Verbal  Information.  The  "facts"  that  comprise  much 
of  the  information  passed  on  in  an  instructional 
environment.  Alone  they  have  little  meaning,  but  they 
provide  direction  to  learning  and  aid  in  the  transfer 
of  learning. 
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3.  Cognitive  Strategies.  Describes  the  cognitive 
ability  of  individuals  to  know  how  and  when  to 
integrate  intellectual  skills  and  verbal  information. 

4.  Hotor  Skills.  Involves  the  physical  act  of  human 
performance  ranging  from  writing  to  steering  an 
automobile . 

5.  Attitudes.  Attitudes  can  often  influence  the 
choice  of  action  of  an  individual. 

(Gagne  and  Briggs,  1979:49-55;  Goldstein,  1984:99) 

To  be  effective,  the  objectives  of  instruction  (that 
dictate  the  method)  need  to  consider  the  different  levels  of 
learning  outcomes  identified  above  by  breaking  up  a  total 
task  into  distinct  components  that  are  defined  by  the  five 
types  of  learning  (Goldstein,  1984:65).  A  more  effective 
transfer  of  learning  from  instructional  setting  to  the  work 
environment  will  result  if  tasks  are  broken  out  and  properly 
defined  and  uniquely  taught.  In  Gagne  and  Briggs' 

Principles  of  Instructional  Design  (1979),  the  use  of  the 
taxonomies  developed  by  Bloom  and  Krathwohl  described 
earlier  are  referenced  as  useful  tools  for  this  purpose. 

Integrating  the  ta.^ks  required  of  a  particular  job  only 
covers  one  aspect  of  the  training  environment.  Gagne  and 
Briggs  (1979)  also  identify  design  instruction  as  part  of 
the  effort  to  emphasize  learning  outcomes  as  the  goals  of  an 
instructional  system  and  bring  a  knowledge  of  the  learning 
process  to  bear  on  the  design  of  instruction.  This  concept 
is  known  as  the  systems  approach. 

The  Systems  Approach.  A  systems  approach  to  training 
instruction  is  identified  as  the  most  effective  method  for 
training  development  and  evaluation  (Goldstein,  1974:17; 
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Ford  and  Wroten,  1984:651;  Carnevale  and  Schultz,  1990:8-16) 
and  involves  the  integration  of  organizational  goals  and 
training  objectives.  Essentially,  the  entire  training 
program  is  an  instructional  system  driven  by  the  assessment 
of  an  instructional  need  within  the  organization.  This  need 
is  then  used  to  identify  behavioral  objectives  that 
emphasize  matching  “training  environments  to  required 
behaviors."  (Goldstein,  1974:22) 

Instructional  Systems  Development  Process.  The 
Instructional  Systems  Development  (ISD)  Process  defines  a 
specific  systems  approach  to  training  development  that 
includes  evaluation.  Gagne  and  Briggs  (1979)  define  the  ISD 
process  as  a  series  of  iterative  and  cyclical  stages  that 
define  an  instructional  system  first  at  the  system  level, 
then  course  level,  and  finally  lesson  level  (20-23). 

Its  structured  methodology  makes  it  well  suited  for  Air 
Force  use.  Air  Force  requirements  for  an  instructional 
system  to  equip  members  to  perform  a  wide  variety  of  jobs 
led  to  the  development  of  its  first  major  instructional 
system  in  1965.  The  evolution  of  that  system  which  covers  a 
broad  range  of  students,  course  lengths  and  subjects  has 
produced  the  Air  Force  ISD  model.  Demonstrated  gains  in  the 
efficiency  of  instruction  that  have  resulted  from  this  model 
has  led  to  Air  Force  policy  that  directs  the  use  of  the  ISD 
process  (AFM  50-2:4). 

The  USAF  ISD  Model .  The  Air  Force  defines  the  ISD 
process  as  A  deliberate  and  orderly,  but  flexible,  process 
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for  planning  and  developing  instructional  programs  which 
ensure  that  personnel  are  taught  in  a  cost  effective  way  the 
knowledges,  skills,  and  attitudes  essential  for  job 
performance  (AFM  50-2:74).  This  process  is  implemented  in 
the  form  of  a  five  step  model  identified  in  Figure  5  that 
includes  the  following  areas: 

1.  Analyze  System  Requirements.  Involves  determining 
what  the  skilled  performer  does  while  doing  the  job, 
how  well  he  or  she  must  do  it,  and  under  what 
conditions . 

2.  Define  Education/Training  Requirements. 

Determining  if  instruction  is  needed  and,  if  so, 
determining  what  instruction  to  give  the  untrained 
personnel  so  they  can  do  the  job  well. 

3.  Develop  Objectives  and  Tests.  Involves  expressing 
instructional  needs  as  specific  objectives  and  devising 
test  items  for  objectives  to  determine  if  they  are  met. 

4.  Plan,  Develop,  and  Validate  Instruction.  Includes 
the  selection  of  media  and  methods,  and  also  developing 
validating  and  revising  the  instructional  materials. 

5.  Conduct  and  Evaluate  Instruction.  Actually 
conducting  the  instruction  and  then  evaluating  the 
program  by  measuring  the  student's  ability  to 
accomplish  the  above  test  items.  (AFP  50-58,  1978:1-5) 

ISP  Model  Application  to  CAT  Leader  Training.  Research 

into  the  history  of  CAT  leader  training  has  not  indicated  a 

comprehensive  systems  approach  in  the  course  development-- 

which  would  include  plans  for  evaluation.  However,  although 

no  complete  instructional  system  was  initially  set  up,  the 

established  curriculum  and  syllabus  allow  the  use  of  the  ISD 

model  in  CAT  leader  training  evaluation  and  improvement. 

As  noted  earlier,  specific  behavioral  objectives 

(system  requirements)  were  identified  relating  to  the 
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CURRKXIIJLIM  UX)P 


FEEDBACK  AND  INTERACTION  LOOP 


Figure  5.  Instructional  Systems  Development  Model  (AFP 
50-58:1-5) 


transfer  of  learning  to  the  organization  (ASD/TQ,  undated: 
Tab),  these  system  requirements  led  to  a  course  notebook  and 
syllabus  (training  requirements)  that  have  been  established 
and  implemented.  This  study  constitutes  addressing  the 
final  three  steps  in  the  ISD  process  that  include  testing, 
validating  and  evaluating  instruction. 

In  chapter  1,  the  research  scope  was  identified  to 
include  mainly  evaluation  of  the  current  syllabus.  Specific 
emphasis  regarding  analyzing  system  requirements  (step  1), 
and  defining  training  requirements  (step  2)  are  left  for 
future  efforts.  This  leads  to  course  evaluation  being  the 
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primary  issue  of  this  paper.  The  next  section  provides  more 
background  relating  to  this  requirement. 

Training  Evaluation 

This  section  considers  issues  related  to  the  evaluation 
of  a  training  program.  The  following  paragraphs  identify 
several  underlying  issues  that  relate  to  training  evaluation 
and  are  designed  to  provide  a  general  overview  of  evaluation 
approaches.  Goldstein  (1984)  was  found  to  provide  the  most 
comprehensive  view  of  training  evaluation  found  and  is  a 
highly  encouraged  reference  for  work  in  this  area. 

The  Need  for  Training  Evaluation.  Training  evaluation 
is  an  issue  that  is  highly  encouraged  by  most  organizations, 
but  seems  to  draw  very  little  support  in  application. 

(Bunker  and  Cohen,  1988:4;  Fisher  and  Weinberg,  1988:73). 
Reasons  to  evaluate  are  overshadowed  by  the  many  reasons 
given  not  to  evaluate,  or  at  least  to  do  a  cursory  job. 

These  reasons  often  include  the  fact  that  programs  were 
developed  by  experts  (Bell  and  Kerr: 70;  Bunker  and  Cohen, 
1988:4),  that  good  evaluation  is  too  expensive  (Zenger  and 
Hargis,  1982:10;  Bunker  and  Cohen,  1988:73;  Fisher  and 
Weinberg,  1988:73)  or  significant  findings  will  not  result 
(Arvey,  et  al . ,  1985:494).  Unfortunately  several  sources 
identified  fear  of  negative  findings  as  another  main  reason 
for  lack  of  evaluation  (Bunker  and  Cohen,  1988:4;  Fisher  and 
Weingberg,  1988:73). 
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The  perceived  drawbacks  to  an  evaluation  program  need 
to  be  examined  with  respect  to  advantages.  The  need  to 
justify  productive  training  as  well  as  reduce  the  amount  of 
ineffective  training  provides  a  strong  basis  to  support 
evaluation  (Zenger  and  Hargis,  1982:11;  Bell  and  Kerr, 
1987:70;  Bunker  and  Cohen,  1988:5;  Fisher  and  Weinberg, 
1988:73)  as  does  the  emphasis  on  evaluation  in  the 
development  of  instructional  systems  and  the  systems 
approach  to  training  (Goldstein,  1984;  Gagne  and  Briggs, 
1987). 

Along  with  the  justification  for  evaluation  is  the 
identified  availability  of  standard  research  methodology 
that  allows  for  meaningful  evaluation  of  programs  (Zenger 
and  Hargis,  1982:11;  Zammit,  1987:262;  Bunker  and  Cohen, 
1988:4) . 

Summative  vs  Formative  Evaluation.  Often  when 
evaluation  is  considered,  the  focus  is  on  an  established 
system  and  just  looking  at  the  results  it  produces. 
Evaluation  involves  much  more.  To  be  comprehensive, 
evaluation  must  include  the  learner,  the  teacher,  and  the 
instructional  design  used  to  both  develop  and  run  the  course 
(Knirk  and  Gustafson,  1986:215). 

Formative  evaluation  focuses  on  the  development  of  the 
course  and  then  on  improvements.  It  often  is  used  to 
determine  if  the  training  program  is  operating  as  designed 
(Goldstein,  1984:148).  Summative  evaluation,  on  the  other 
hand,  involves  the  appraisal  of  an  established  program 
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through  investigating  the  effectiveness  of  instruction. 
Ideally,  the  use  of  both  evaluations  on  a  program  would 
provide  a  good  evaluation  format.  A  formative  evaluation 
would  look  at  how  close  the  program  being  implemented  is  to 
its  design,  and  it  would  also  investigate  areas  of 
improvement  in  instructional  design.  This  would  provide  a 
firm  basis  for  a  summative  evaluation  to  determine  if  stated 

course  objectives  are  met. 

In  the  summative  area,  several  levels  of  criteria  for 
evaluation  are  identified  and  are  useful  in  defining  the 
level  of  detail  regarding  training  effectiveness  and 
transfer.  These  four  levels  are  defined  below. 

Kirkpatrick's  Four  Levels  of  Criteria  for  Evaluation. 
This  model  is  used  extensively  by  authors  when  considering 
evaluating  training  programs  (Goldstein,  1974;  Kirkpatrick, 
1978;  Ford  and  Wroten,  1984;  Carnevale  and  Schultz,  1990). 
Kirkpatrick's  four  levels  of  criteria  for  evaluation  include 
reaction,  learning,  behavior  and  results.  Weeks  (1986) 
provides  a  good  summary  of  the  criteria: 

1.  Reaction.  Measures  the  degree  to  which  people  are 
satisfied  with  the  training  program.  It  often  provides 
feedback  on  how  course  content,  methodology,  materials 
and  instructors  were  received  by  participants,  but  does 
not  measure  learning  (emphasis  added). 

2.  Learning.  Involves  assessing  whether  or  not 
trainees  have  acquired  any  new  knowledges,  skills  or 
abilities  during  the  classroom  experience. 

3.  Behavior.  Involves  a  measurement  of  a  trainees 
performance  at  a  job  site.  It  attempts  to  identify  and 
document  changes  in  behavior  which  can  be  attributed  or 
related  to  training  experiences. 
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4.  Results.  Involves  measuring  the  impact  of  training 
activities  to  results  regarding  organizational  goals 
and  changes.  (Weeks,  1986:22-23) 

As  indicated  earlier,  the  ultimate  issue  involves  the 
transference  of  learned  behavior  from  the  classroom  to  the 
work  environment.  Although  the  desired  evaluation  to 
measure  effectiveness  would  continually  point  towards 
Kirkpatrick's  Level  4  evaluation  criteria  (Results),  that 
summative  level  of  assessment  is  relatively  difficult  to 
obtain.  The  result  is  a  reliance  on  the  lower  levels  of 
evaluation  that  are  easier  to  develop  and  implement,  but 
provide  less  information  (Weeks,  1986:23). 

Goldstein  identified  a  Catalanello  and  Kirkpatrick 
study  which  found  that  out  of  154  companies  surveyed,  77 
percent  stressed  evaluation  studies  related  to  reactions 
(Goldstein,  1974:61).  Prior  to  this  research  effort,  CAT 
leader  training  fell  into  this  category  of  evaluation  since 
the  only  substantive  evaluation  involved  review  of  critique 
forms  provided  at  the  end  of  the  course. 

Part  of  the  problem  for  the  reaction  level  emphasis  is 
the  level  of  difficulty  in  higher  level  assessments  and  the 
large  number  of  factors  beyond  classroom  training  that 
affect  both  job  performance  and  organizational  results 
(Weeks,  1986:23). 

Research  design  is  a  major  issue  that  relates  to  many 
of  the  factors  that  can  confound  an  evaluation.  The 
selection  of  a  design  and  methodology  and  the  implementation 
of  the  research  provide  a  means  to  effectively  meet 
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evaluation  objectives  of  time,  money,  and  information.  The 
next  section  discusses  the  use  of  a  research  design  in 
training  evaluation. 

Research  Design 

Research  design  discussions  were  prominent  in  training 
evaluation  literature  (Goldstein,  1974;  Bunker,  1978; 
Kirkpatrick,  1978;  Zenger  and  Hargis,  1982;  Zammit,  1987; 
Carnevale  and  Schultz,  1990).  Almost  all  discussions 
involved  the  use  of  designs  proposed  by  Campbell  and  Stanley 
(1963).  These  include  pre-experimental  designs,  true 
experimental  designs,  and  quasi-experimental  designs.  The 
use  of  pre-  and  post-testing  was  also  highly  recommended  for 
evaluating  learning  in  a  training  program,  as  was  the  use  of 
control  groups  (Goldstein,  1974;  Bunker  and  Cohen,  1978; 
Zenger  and  Hargis,  1982;  Kirkpatrick,  1978;  Carnevale  and 
Schultz,  1990). 

Research  rigor  does  come  at  a  cost,  and  requirements  or 
resources  must  be  considered  when  reviewing  possible 
evaluation  approaches.  No  more  rigorous  design  should  be 
used  than  required  (Kirkpatrick,  1978;  Carnevale  and 
Schultz,  1990). 

Often  a  main  criticism  of  research  deals  with  the  issue 
of  validity.  In  Mitchell's  article  An  Evaluation  of  the 
Validity  of  Correlational  Research  Conducted  in 
Organizations  (1985),  one  anonymous  respondent  to  a 
Campbell,  Daft,  and  Hulin  paper  (1982)  was  quoted  describing 
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the  major  concern  of  using  very  powerful  and  computer- 
related  statistical  techniques  to  draw  conclusions  from 
questionable  measurement  instruments  (Mitchell,  1985:192). 
Clearly,  any  research  conclusions  need  to  be  considered  with 
respect  to  the  validity  of  the  instrvunents  and  research 
methods . 

Internal /External  Validity.  Evaluation  is  mainly 
directed  at  two  specific  issues:  internal  and  external 
validity.  Internal  validity  involves  the  degree  of  in-class 
learning  that  results  from  training  and  external  validity 
involves  the  accuracy  of  applying  training  behavior  to 
actual  on-the-job  performance  (Bunker  and  Cohen,  1978). 

Content  Validity.  Content  validity  refers  to  the 
applicability  of  training  related  criteria  to  on-the-job 
application  and  addresses  some  aspects  of  external  validity. 

A  quantitative  approach  to  assess  job-relatedness  was 
developed  by  Lawshe  (1975)  and  involved  measuring  the 
content  validity  of  individual  test  items  of  a  selection 
test  by  having  "subject  experts"  rate  each  test  item  as  it 
was  applicable  to  a  specific  job. 

The  Content  Validity  Ratio  (CVR)  rates  training 
specific  knowledges,  skills,  abilities  and  other  personal 
characteristics  (KSAOs)  as  having  job  application.  The 
usefulness  of  this  approach  involves  two  assumptions.  The 
first  assumption  is  if  more  than  50%  of  the  "subject 
experts"  rate  a  particular  KSAO  as  important,  it  has  some 
degree  of  job  relatedness.  The  second  assumption  is  that 
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the  more  experts  who  rate  it  as  important,  the  greater 
degree  of  content  validity  (Lawshe,  1975;  Ford  and  Hroten, 
1984). 

The  following  formula  describes  the  linear 
transformation  that  defines  the  CVR: 


CVR- 


Wf-Ni 


Where  is  the  number  experts  who  rate  a  task  as 
essential  and  Nj  is  the  total  number  of  raters. 

Each  training  item  is  identified  as  a  task  that  will  be 
rated  by  the  content  experts  as  either  essential  (1),  useful 
but  not  essential  (0),  or  not  necessary  (-1). 

The  CVR  was  originally  introduced  to  aid  in  identifying 
content  valid  test  items  (Lawshe,  1975).  Its  application  to 
training  evaluation  was  identified  by  research  regarding 
police  recruit  training  (Ford  and  Hroten,  1984).  Several 
research  applications  have  modified  the  CVR  approach  to 
measure  content  validity  of  a  training  post-test  (Distefano, 
et  al . ,  1980),  the  content  validity  of  training  elements 
(Ford  and  Hroten,  1984),  and  the  content  validity  of 
behavioral  rating  scale  criterion  (Distefano,  et  al . ,  1983). 

The  application  of  the  content  validity  ratio  to 
training  related  areas  has  demonstrated  the  usefulness  of 
this  procedure  in  supporting  the  content  validity 
conclusions  in  several  research  efforts. 
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Summary 

This  chapter  reviewed  the  general  theories  of  learning 
and  several  aspects  of  the  instructional  system  design 
approach.  Since  this  research  involves  the  evaluation  of  a 
training  program,  emphasis  was  placed  on  this  area  of 
instructional  system  design  and  included  specific  literature 
dealing  with  training  evaluation  and  research  design. 

Chapter  4  will  now  take  much  of  the  information  just 
discussed  and  provide  a  methodology  that  focuses 
specifically  on  the  research  accomplished  for  CAT  leader 
training . 
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IV.  Method 


Introduction 

This  chapter  discusses  the  methods  used  in 
accomplishing  the  research  objectives.  It  describes  the 
identification  of  a  training  approach  and  the  development  of 
behavioral  objectives.  These  objectives  were  the  basis  for 
developing  a  knowledge  test  for  an  internal  course 
evaluation,  and  were  also  used  for  developing  a 
questionnaire  for  a  field  evaluation.  Statistical  analysis 
was  used  on  the  data  collected  to  answer  the  majority  of  the 
research  objectives. 


Restatement  of  the  Research  Objectives 

The  following  objectives  identified  in  Chapter  1  form 
the  basis  for  the  research  direction.  For  reference,  the 
research  objectives  are  again  provided: 

1.  Identify  and  document  the  specific  material 
provided  as  part  of  the  current  course  curriculum. 

2.  Determine  if  the  CAT  leader  training  course 
facilitates  learning  of  the  course  material  and 
improves  a  trainee's  self-efficacy  regarding 
application  of  course  material. 

3.  Determine  the  characteristics  of  current  corrective 
action  teams  and  the  involvement  of  graduates. 

4.  Identify  the  level  of  self-efficacy  of  past 
attendees  regarding  application  of  course  material. 

5.  Identify  how  past  attendees  perceive  and  apply  the 
problem  solving  tools  and  technigues  provided  in  the 
course . 
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6.  Based  on  the  above  objectives,  provide  the 
Commander  of  ASD  with  recommendations  for  improvements 
to  the  current  CAT  leader  training  curriculum. 

General  Research  Direction 

The  general  research  method  involved  using  data  from 
several  sources  to  address  the  research  objectives  and 
answer  the  research  questions.  Since  only  past  course 
critiques  were  available  on  the  training  to  date,  secondary 
data  sources  were  not  available,  making  primary  data  the 
principal  source  of  information. 

The  Commander's  Office  for  Total  Quality  (ASD/TQ)  is 
the  host  of  the  ASD  CAT  leader  training  program  and 
sponsored  the  course  evaluation  research.  They  made 
available  all  necessary  resources  for  support.  For  data 
collection,  ASD/TQ  supported  an  in-class  evaluation  of  four 
courses  and  provided  attendance  rosters  for  all  previous 
classes  that  identified  the  course  population  of  440 
individuals . 

The  following  paragraphs  identify  the  specifics 
associated  with  the  effort. 

Identifying  Training  Objectives 

Currently,  CAT  leader  training  has  three  overall 

objectives  that  were  first  identified  in  Chapter  2.  They 

include  training  a  CAT  leader: 

1.  To  be  able  to  organize  a  corrective  action  team  and 
manage  the  group  dynamics  so  that  the  CAT  develops  into 
and  remains  an  effective,  functioning  team. 
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2.  To  be  able  to  explain  the  TQ  structure  and 
philosophy  to  individual  members  and  how  the  CAT  fits 
into  the  overall  TQ  process. 

3.  To  be  able  to  teach  the  problem  solving  methodology 
and  the  appropriate  tools  and  techniques  to  CAT  members 
so  that  they  are  able  to  effectively  use  them  in  the 
problem  solving  process.  (ASD/TQ,  vindated:Tab) 

These  objectives  explain  the  goals  of  the  training  well 

as  it  relates  to  ASD  and  could  properly  be  named  "terminal 

objectives,"  because  they  identify  what  a  CAT  leader  should 

be  able  to  accomplish  after  completion  of  the  program 

(Goldstein,  1974:21).  However,  to  measure  the  terminal 

objectives  by  a  test  instriiment,  the  identification  of 

subobjectives  is  required  to  measure  individual  criteria  of 

the  training.  Subobjectives  measure  subskills  necessary  to 

master  terminal  objectives  (Zammit,  1987:252).  Since  no 

subobjectives  previously  existed,  the  initial  step  in  the 

research  required  their  development. 

Developing  Specific  Subobjectives.  Both  terminal  and 

subobjectives  (hereafter  both  referred  to  as  "objectives") 

are  behavioral  objectives  that  make  statements  about 

behaviors  and  content  (Zammit,  1987:255). 

Behavior  statements  include  action  verbs  that  identify 

requirements  of  students  (e.g.,  understand,  describe, 

select,  etc.).  Content  statements  involve  specific  subject 

matter,  like  steps  in  a  process.  The  three  behavior 

statements  proposed  for  CAT  leader  training  objectives 

included : 
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1.  Have  an  appreciation  for 

2.  Identify  the  steps,  functions,  diagrams  and 

definitions 

3.  Know  the  principles  of 

The  content  statements  proposed  for  CAT  leader  training 
objectives  included: 

1.  The  steps  in  a  technique. 

2.  The  stages  in  a  model. 

3.  the  steps  in  a  process. 

These  statements  were  used  to  develop  objectives 
relating  to  the  CAT  leader  training  blocks  identified  in 
Figure  2.  These  blocks  included:  Overview  of  Total  Quality 
Management,  Effects  of  Personal  Values,  Group  Dynamics,  the 
Problem  Solving  Process,  Problem  Solving  Tools  and 
Techniques  and  Meeting  Effectiveness.  The  CAT  leader 
notebook  provided  to  all  attendees  since  training  began  in 
1989  was  thoroughly  reviewed  to  generate  a  comprehensive 
list  of  proposed  course  objectives.  Recommendations  for 
additional  objectives  not  identified  were  requested  during 
the  review  cycle. 

Review  of  Proposed  Objectives.  Objectives  developed 
from  the  content  of  the  training  blocks  described  above  were 
initially  provided  in  the  form  of  a  questionnaire  to  twelve 
employees  of  ASD  with  recent  CAT  leader  training  experience. 
They  included  four  past  course  instructors  and  eight  recent 
attendees.  Since  the  objectives  were  to  mirror  the  current 
course  (and  not  to  identify  improvements),  reviewers  were 
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instructed  to  make  comments  related  only  to  the  current 
course  curriculum.  They  were  also  asked  to  apply  the 
following  criteria  to  their  review: 

1.  Are  all  objectives  applicable  to  the  training? 

2.  Is  the  scope  of  required  knowledge  (have  an 

appreciation  for,  identify,  and  know)  for  each 

objective  appropriate? 

3.  Are  there  missing  objectives? 

4.  Are  the  objectives  clear  and  concise? 

Comments  were  collected  and  an  updated  set  of  proposed 
objectives  (with  changes  noted)  were  again  distributed  to 
the  same  individuals.  In  addition,  proposed  objectives  with 
the  same  review  guidelines  were  also  provided  to  several 
experts  in  organizational  research  knowledgeable  in  training 
programs . 

Format  and  content  changes  were  reviewed  and 
incorporated  from  the  comments  received.  Objectives  were 
categorized  into  training  blocks  as  they  are  organized  in 
the  CAT  leader  notebook  and  the  course  syllabus  (ASD/TQ, 
undated : Syl 1 abus ) .  A  set  of  developed  course  objectives  is 
provided  in  Appendix  B.  Only  objectives  dealing  with  the 
current  course  curriculum  were  allowed. 

Content  Validity  of  the  Objectives.  Since  a  validated 
set  of  objectives  is  required  to  ensure  the  development  of  a 
reliable  knowledge  test  and  questionnaire,  it  was  necessary 
to  ensure  that  actual  instruction  addressed  those 
objectives.  To  accomplish  this,  questionnaires  were 
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provided  to  the  12  members  of  the  current  course  instructor 
cadre.  The  instructors  were  asked  to  review  the  objectives 
and  rate  each  on  the  scale  identified  in  Figure  6  as  they 
relate  to  their  own  presentations: 


* 

(1) 

I 

completely  cover  this  objective 

(2) 

I 

partially  cover  this  objective 

(3) 

I 

do  not  cover  this  objective 

(4) 

I 

have  never  taught  this  section  of  the  course 

Figure  6.  Instructors  Options  to  Identify  What 

Objectives  are  Taught  in  CAT  Leader  Training 
(modified  from  Lawshe,  1975) 


Since  the  course  is  team-taught,  option  four  was 
included  to  account  for  those  instructors  who  have  never 
presented  a  particular  training  block. 

Nine  of  12  instructors  returned  the  objectives  rating 
form  (including  the  four  facilitators  who  normally  conduct 
the  training).  The  rating  forms  were  scanned  and  the  raw 
data  analyzed  using  a  modified  version  of  Lawshe's  (1975) 
Content  Validity  Ratio  (CVR)  technique  discussed  in  Chapter 
3.  Specific  objectives  were  considered  to  be  completely 
covered  in  class  by  all  instructors  if  the  corresponding  CVR 
was  greater  than  or  equal  to  an  established  minimum  CVR 
standard  (p  <  .05).  This  standard  value  is  drawn  from 
Schippers'  computations  that  considered  both  the  confidence 
desired  and  the  number  of  reviewers  (Lawshe,  1975:568). 
Because  of  the  small  sample  size  (N>9),  CVR  frequency 
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calculations  were  performed  using  the  software  package 
Statistix^*  on  a  386SX  personal  computer. 

The  values  of  individual  CVRs  for  objectives  are 
located  in  Appendix  C  and  form  the  basis  for  answering 
investigative  question  1  that  relates  to  the  curriculum 
taught  by  course  facilitators.  Since  the  proposed 
objectives  were  the  basis  for  the  knowledge  test,  the 
results  of  the  content  validity  calculations  became  useful 
both  to  identify  valid  course  objectives  and  validate  test 
items.  A  discussion  of  the  knowledge  test  development  is 
provided  in  the  following  section. 

Development  of  Knowledge  Test 

The  knowledge  test  was  the  primary  instrument  for  the 
internal  evaluation  of  the  course.  It  was  designed  as  a 
suiranative  evaluation  instrviment  to  assess  the  effectiveness 
of  the  course  design  and  delivery  (Zammit,  1987:252),  and 
address  research  objectives  2,  3  and  4. 

A  knowledge  test  is  a  criterion-referenced  test  that 
measures  specific  behavioral  objectives  (Goldstein, 
1984:134).  It  involves  pre-testing  and  post-testing  the 
training  subjects  to  assess  how  well  students  master  the 
objectives  developed  for  a  course.  As  discussed  in  the 
scope  of  the  research  project  in  Chapter  1,  the  knowledge 
test  involved  the  knowledge  and  comprehensive  levels  of 
learning  (Bloom,  et.  al ,  1971).  As  a  result,  the  questions 
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were  developed  from  those  objectives  where  the  requirement 
was  to  know  or  be  able  to  identify  course  material. 

Testing  Design.  Variables  measured  during  testing  are 
associated  with  the  behavioral  learning  objectives  of 
training  (Goldstein,  1974:73),  and  involve  the  following 
experimental  design  format: 


Where  Oj  was  the  pre-test,  X  the  training,  and  the  post¬ 
test  (Campbell  and  Stanley,  1963:7).  This  design  is  a 
derivative  of  the  pre-test/post-test  pre-experimental  design 
commonly  used  in  instructional  settings  that  provides  a 
measure  of  comparison  to  the  same  group  of  subjects 
(Goldstein,  1974:79).  The  control  group  involved  one  class 
of  subjects  who  took  only  the  post-test  after  training,  but 
not  the  pre-test.  Data  was  collected  from  this  group  to 
indicate  effects  of  student  questions  prompted  by  the  pre¬ 
test.  The  short  time  duration  of  training  should  limit  the 
internal  validity  threats  due  to  maturation  and  history. 
However,  the  impact  of  pre-test  sensitizing  on  post-test 
scores  is  a  concern. 

Test  Writing.  Questions  were  placed  in  two  separate 
sections  of  the  knowledge  test  located  in  Appendix  D. 

Section  1  involved  personal/organizational  topics  and 
collected  demographic  data  and  the  student’s  past  experience 
with  teams  and  training  associated  with  portions  of  the 
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course  curriculum.  At  the  end  of  this  section,  5  questions 
related  to  self-efficacy  were  also  provided.  These  self- 
efficacy  questions  were  again  repeated  in  the  questionnaire 
given  to  past  course  attendees. 

Section  2  of  the  exam  addressed  CAT  leader  training 
material  identified  by  the  training  blocks  in  Figure  5. 
Questions  were  developed  based  on  those  objectives  that 
required  a  ''knowledge  of"  or  an  "appreciation  for"  the 
material.  Sixty-three  questions  were  written  by  the 
researcher  under  the  following  guidelines: 

1.  Ensure  each  test  item  matches  an  objective. 

2.  Ensure  all  information  required  for  an  appropriate 

response  is  provided. 

3.  Ensure  each  test  item  is  clearly  stated. 

4.  Ensure  only  relevant  material  is  provided. 

5.  Review  all  test  items  for  content. 

6.  Have  instrument  reviewed  and  pretested. 

(Zammit,  1987:257;  Emory,  1985:228} 

Guidelines  1,  2  and  4  involved  question  content,  2  and 
3  involved  question  wording,  and  5  and  6  involved  item 
review  (Zamnit,  1987:257). 

The  test  was  reviewed  by  Air  Force  Institute  of 
Technology  (AFIT)  faculty  who  are  experts  in  testing  and 
organizational  theory  and  included  issues  such  as:  test 
question  content  applicability  to  behavioral  objectives,  the 
presence  of  unstated  or  misleading  assumptions,  the  presence 


50 


of  wording  bias,  and  question  scope  and  clarity  (Emory, 
1985:228). 


Since  the  knowledge  test  was  written  prior  to 
validating  behavioral  objectives,  sixty-three  questions  were 
developed  for  the  test  instrument.  Of  those  sixty- three 
questions,  seven  were  related  to  objectives  which  had 
negative  CVR  values.  The  results  of  these  items  were  not 
considered  in  any  assessment  of  student  performance. 

Pre-testing.  Pre-testing  was  accomplished  with  ten 
APIT  graduate  students  to  identify  test  question  clarity  and 
completeness  and  also  ease  of  answering  (Emory,  1985:207). 
The  students  were  selected  based  on  availability  and  were 
screened  to  ensure  they  had  not  previoxisly  received  CAT 
leader  training.  Current  CAT  leader  training  instructors 
were  not  included  in  the  review  to  preclude  their  bias 
toward  teaching  test  material  during  data  collection. 

Improvements  were  made  to  the  exam  that  included 
increasing  the  age  options  available  for  demographics, 
highlighting  negative  response  questions,  and  providing  more 
of  a  random  order  to  the  matching  portions  of  the  test. 

Knowledge  Test  Data  Collection.  Each  exam  began  with 
an  administrator  providing  instructions  to  the  students. 

Four  classes  were  tested  in  consecutive  months.  A  post-test 
was  given  to  the  control  group  in  March,  followed  by  three 
groups  receiving  both  the  pre-test  and  post-test  set  in 
April,  May,  and  June. 


The  researcher  administered  the  test  for  the  first  two 
classes  and  provided  instructions  for  the  third.  The  two 
instructors  who  facilitated  all  three  classes  tested  were 
provided  instructions  and  administered  the  third  exam. 

Each  participant  received  an  exam,  an  optical  score 
sheet  (AFIT  Form  llD) ,  and  a  #2  lead  pencil.  In  all  cases, 
the  classes  were  told  the  general  scope  of  the  research  and 
it  was  stressed  that  individual  raw  scores  were  not  an 
important  part  of  the  research  requirements.  These  steps 
were  taken  to  limit  threats  to  validity  by  participants  who 
might  interpret  the  knowledge  test  as  a  source  of  criticism 
or  grading  (Bloom,  et  al . ,  1971).  The  first  set  of  data 
collected  involved  the  post-test  group  that  completed  CAT 
leader  training  on  29  March  1991.  Testing  results  coded  on 
optical  scan  sheets  were  read  into  a  file  for  analysis. 

Knowledge  Test  Population.  The  post  test  control  group 
consisted  of  17  students  from  the  Special  Forces  System 
Program  Office  (ASD/VX).  They  contained  a  slightly  higher 
proportion  of  middle  management,  but  still  covered  all 
ranges  of  personnel.  The  population  tested  both  before  and 
after  training  consisted  of  forty-four  students  in  three 
classes  from  organizations  throughout  ASD.  Nine  students 
were  military  a  third  of  whom  were  enlisted,  with  officers 
representing  the  ranks  of  lieutenant  through  major.  Thirty- 
two  civilians  were  included  in  groups  ranging  from  GS-03  to 
GM-14.  Three  contractors  attended.  The  median  age  of  all 
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subjects  was  36-40  years  with  a  nearly  equal  representation 
of  both  men  and  women. 

Knowledge  Test  Reliability.  Reliability  involves 
determining  how  well  an  instrument  can  consistently  produce 
the  same  results  over  time.  High  reliability  indicates  that 
variance  in  the  scores  is  not  due  to  chance  arising  from  a 
flawed  measuring  instrument  (Guion,  1965:27). 

Reliability  calculations  were  used  to  determine 
Cronbach's  alpha  coefficient  that  is  interpreted  as  the 
average  of  all  possible  split-half  reliabilities.  Using  the 
Fifty-five  test  items  developed  from  validated  objectives 
provided  a  Cronbach  alpha  coefficient  of  0.89,  indicating  a 
good  to  excellent  reliable  test  instrument  (Shane,  1991). 
Review  of  the  frequency  distributions  for  the  questions 
showed  a  range  of  responses  for  each  test  item  that  gave  no 
indication  of  overly  easy  or  poorly  written  test  questions. 
Additionally,  no  significant  changes  in  reliability  were 
identified  when  individual  items  were  removed  and 
reliability  re-calculated. 

Based  on  the  promising  post-test  results,  three 
experimental  groups  receiving  CAT  leader  training  during  the 
months  of  April,  May,  and  June  were  given  the  knowledge  test 
on  both  the  first  and  last  days  of  training.  These  three 
groups  were  selected  to  improve  statistical  power.  In 
addition  to  an  increased  sample  size  for  statistical 
purposes,  several  groups  were  selected  to  measure  any 
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effects  that  result  from  a  particular  instructor.  However, 
measuring  the  effects  of  particular  instructors  was  not 
possible  since  the  same  instructors  actually  taught  all  the 
classes  tested.  These  instructors  reviewed  the  course 
objectives,  but  were  not  exposed  to  the  test  instrument. 

Each  class  took  a  pre-test  on  the  Tuesday  morning 
opening  of  the  course  and  concluded  Friday's  course 
requirements  with  the  same  knowledge  test  now  representing  a 
post-test.  The  Cronbach's  alpha  coefficients  are  identified 
in  Table  1  for  each  of  the  sets  of  tests.  As  shown,  each 
course  reliability  figure  indicates  a  highly  reliable  test 
instrument . 


Table  1 

Knowledge  Test  Reliability 


Type  of  test 

N 

Dates 

Cronbach's 

Alpha 

Post-test 

17 

29  Mar  91 

.89 

Pre-test 

44 

9  Apr  91 

.85 

14  May  91 

11  Jun  91 

Post-test 

44 

12  Apr  91 

.89 

17  May  91 

14  Jun  91 

Development  of  CAT  Leader  Training  Questionnaire 

The  questionnaire  was  the  primary  instrument  for  the 
field  evaluation.  It  was  designed  to  address  objectives  3, 
4  and  5  of  the  research  in  determining  the  extent  that 
training  material  is  applied  and  how  it  is  perceived  by  CAT 
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leaders  in  the  ASD  environment.  Since  there  was  a 
reasonably  small  population  of  past  attendees  (440 
students),  the  use  of  a  short  questionnaire  was  viewed  as 
the  most  efficient  and  economical  option  to  collect  the  data 
to  address  the  objectives  (Emory,  1985:158).  Confidential 
responses  and  guidelines  required  by  the  Privacy  Act  of  1973 
also  promoted  an  honest  response  to  questionnaire  items. 

Testing  Design.  Variables  measured  during  field 
evaluation  are  associated  with  the  transfer  of  the 
behavioral  learning  objectives  of  training  (Goldstein, 
1974:73).  This  testing  involves  a  pre-experimental  design 
of  a  one-shot  case  study  having  the  format: 

X  O2 

Where  X  is  the  training,  and  0^  is  the  questionnaire 
(Campbell  and  Stanley,  1963:6).  Since  no  previous  data  was 
available,  the  one-shot  case  study  was  chosen  to  collect 
perceptions  from  CAT  leaders.  While  it  is  possible  to 
measure  the  application  of  course  material,  measuring  any 
change  in  attitude  or  perception  is  not  possible  (Weisberg 
and  Bowen,  1977:88).  As  a  result,  this  portion  of  the 
research  is  limited  to  only  providing  a  snapshot  of  material 
application.  The  ability  to  then  conclude  that  the  training 
received  was  the  responsible  factor  leading  to  the  behavior 
change  is  very  difficult  to  support.  However,  future 
research  may  use  this  data  to  draw  such  conclusions. 
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Questionnaire  Population.  The  population  of  interest 


were  those  ASD  employees  who  had  received  CAT  leader 
training.  As  of  May  1991,  440  individuals  have  received  CAT 
leader  training  from  23  different  course  offerings.  While 
many  Air  Force  organizations  have  sent  personnel  to  CAT 
leader  training,  questionnaires  were  distributed  only  to  the 
373  individuals  associated  with  ASD,  the  sponsor  of  the 
research. 

The  population  included  members  of  all  military  and 
civilian  ranks  and  grades.  Military  members  ranged  in  rank 
from  enlisted  to  the  rank  of  colonel  and  civilian  attendees 
included  GS-03  through  GM-15.  Five  contractors  also 
responded  to  the  questionnaire.  The  median  age  of 
respondents  was  between  36-40  years.  Men  represented  almost 
2  out  of  3  respondents. 

Questionnaire  Format.  The  entire  questionnaire  package 
consisted  of  five  parts:  the  cover  letter,  one 
questionnaire,  an  optical  scan  response  form  (AFIT  Form 
lie),  a  list  of  definitions,  and  an  addressed  return 
envelope.  The  package  was  reviewed  and  approved  by  the 
Commander's  Office  for  Total  Quality  (ASD/TQ).  The  cover 
letter  that  introduced  the  questionnaire  was  signed  by  the 
Commander's  Assistant  for  Total  Quality  and  provided  the 
respondent  with  a  brief  background  on  the  project  and 
encouraged  participation. 


56 


The  questionnaire  itself  was  divided  into  six  separate 
sections  identified  in  Figure  7.  Each  section  title  was 
followed  with  a  brief  sumnary  which  identified  to  the 
respondent  the  type  of  information  that  was  being  furnished 
in  that  section.  This  additional  information  was  supplied 
to  further  inform  the  respondents  and  encourage  their 
cooperation  in  completing  the  instrument  (Emory,  1985:158- 
159)  . 


Section 

1: 

Personal /organizational 

Section 

2: 

Training  Information 

Section 

3: 

Assignments  and  Status  of  CATs 

Section 

4 : 

Perceived  Abilities 

Section 

5: 

Use  of  Tools  and  Techniques 

Section 

6: 

General  Comments 

Figure  7.  Sections  of  CAT  Leader  Training 
Questionnaire  (Appendix  E) 


Section  1  requested  general  demographic  information 
regarding  the  characteristics  of  respondents  who  have  taken 
CAT  leader  training.  Section  2  requested  information 
regarding  the  date  of  training  for  the  respondent  and 
whether  that  individual  was  still  in  the  organization  that 
sponsored  their  training  slot.  Section  3  requested 
information  on  the  level  and  type  of  CAT  involvement  and  the 
current  status.  Section  4  measured  each  respondent's  level 
of  self-efficacy  in  regard  to  performing  tasks  related  to 
the  CAT  leader  training  curriculum.  Section  5  provided 
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information  on  the  level  and  frequency  of  use  of  problem 
solving  tools  and  techniques  that  were  introduced  in  CAT 
leader  training.  Since  many  respondents  may  not  have 
recognized  some  of  the  tools  and  techniques  identified,  a 
reference  was  made  to  a  list  of  definitions  provided  as  an 
attachment  in  the  questionnaire  package  (Appendix  F). 

Section  6  solicited  feedback  on  any  of  the  previous  sections 
of  the  questionnaire  and  the  CAT  leader  training  course 
itself . 

Items  were  written  by  the  researcher  using  the 
following  guidelines; 

1.  Ensure  all  information  required  for  an  appropriate 

response  is  provided. 

2.  Ensure  each  test  item  is  clearly  stated. 

3.  Ensure  only  relevant  material  is  provided. 

4.  Review  all  test  items  for  content. 

5.  Have  instrument  reviewed  and  pretested. 

(Zammit,  1987:257;  Emory,  1985:228) 

Guidelines  1  and  3  involved  question  content,  2 
involved  question  wording,  and  4  and  5  involved  item  review 
(Zammit,  1987:257).  The  test  was  reviewed  by  AFIT  faculty 
who  are  experts  in  testing  and  organizational  theory  and 
included  issues  such  as:  the  presence  of  unstated  or 
misleading  assumptions,  the  presence  in  bias  of  wording  and 
question  scope  and  clarity  (Emory,  1985:228). 

Pre-testing .  Pre-testing  was  accomplished  with  three 
CAT  leader  training  facilitators,  the  CAT  leader  training 
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course  director,  the  ASD/TQ  focal  point  on  measurement,  and 
four  employees  of  ASD  with  experience  in  TQM.  A  set  of 
proposed  hypotheses  was  also  provided  with  the  questionnaire 
and  each  test  candidate  was  requested  to  complete  the  survey 
and  review  each  item  for  clarity,  ease  of  answering,  and 
applicability  (Emory,  1985:207). 

Improvements  were  made  to  the  questionnaire  that 
included  adding  section  titles  and  svunmaries,  modifying  the 
training  date  options  from  two  questions  to  three  questions 
and  dividing  the  tools  and  techniques  options  into  three 
categories  (level  of  use,  frequency  of  use,  and  usefulness) 
with  corresponding  changes  in  their  associated  Likert 
scales.  A  general  comments  section  was  also  added  to 
solicit  additional  feedback  from  the  respondents. 

Questionnaire  Data  Collection.  Several  days  prior  to 
questionnaire  distribution,  a  summary  of  the  research  plan 
and  objectives  was  sent  to  all  TQ  coordinators  in  ASD.  The 
goal  was  to  notify  ASD  personnel  associated  with  TQ  about 
the  research  and  encourage  them  to  notify  past  CAT  leader 
training  attendees  to  expect  a  questionnaire. 

The  entire  questionnaire  package  was  mailed  out  via  the 
ASD  internal  mail  system  in  the  first  week  of  May  1991  to 
all  past  attendees  of  CAT  training  in  ASD.  The  mailing  list 
was  developed  from  the  attendance  lists  for  the  training 
classes  and  included  ranks  when  provided.  Each  envelope 
identified  the  course  graduate,  his/her  organization,  the 
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training  date,  and  a  request  to  forward.  The  training  date 
was  provided  on  the  label  so  that  information  could  be 
collected  in  Section  2  and  conclusions  drawn  for  different 
classes.  The  return  envelope  was  addressed  to  ASD/TQ  with 
responses  requested  by  14  June  1991.  This  five-week  time 
period  allowed  for  questionnaire  responses  that  considered 
mail  delivery,  response  time,  and  possible  forwarding  to 
reassigned  ASD  personnel.  As  stated  earlier,  373 
questionnaires  were  distributed.  After  2  weeks,  166  (45%) 
were  returned  with  a  total  after  5  weeks  reaching  222  (60%). 
Of  the  222  returned,  13  were  not  completed  due  to  changes  in 
address,  unavailability  of  the  individual,  and  resignations. 
This  provided  an  overall  response  rate  of  208/373  (56%)  and 
was  considered  an  excellent  response  considering  that  some 
of  the  class  rosters  used  to  develop  the  mailing  list  were 
over  1  1/2  years  old.  Responses  were  received  and  coded 
into  a  data  file.  Questionnaire  responses  were  collected  on 
the  AFIT  Form  IIC  that  allowed  for  optical  scanning  of  the 
data  for  coding  and  analysis.  Additional  comments  were 
collected  in  Section  6  of  the  questionnaire  and  are  provided 
in  Appendix  G. 

Data  Analysis 

Analysis  of  the  data  from  both  the  knowledge  test  and 
questionnaire  was  defined  by  the  requirements  of  the 
specific  research  questions  identified  in  Chapter  1.  As 
described  earlier,  the  content  validity  requirements  for  the 
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objectives  was  established  using  a  modified  version  of 
Lawshe's  (1975)  content  validity  ratio  (CVR)  and  content 
validity  index  (CVI).  This  provided  a  basis  for  the 
credible  analysis  of  the  knowledge  test  and  questionnaire 
data. 

Both  sets  of  data  were  collected  on  optical  scan  sheets 
(AFIT  Forms  IID  and  IIC  respectfully)  with  calculations 
being  handled  on  a  VAX  6420  computer  using  the  Statistical 
Analysis  System  (SAS)  software  (version  6.0).  Table  2 
identifies  the  11  research  questions  and  the  associated 
statistical  tests  that  were  required  to  address  each 
question. 


Table  2 

Statistical  Tests  for  Investigative  Questions 


Question 

Measurement  Instrument 

Test 

1 

Training  Objectives 

CVR/ CVI 

2 

Knowledge  test 

Paired  T-test 

3 

Knowledge  test 

Simple  Regression 

4 

Knowledge  Test 

Paired  T-test 

5 

Questionnaire 

Chi-square 

6 

Questionnaire 

Chi-square 

7 

Questionnaire 

Ind-Sample  T-test 

8 

Questionnaire 

Chi-square 

9 

Questionnaire 

Chi-square 

10 

Questionnaire 

Chi-square 

11 

Questionnaire 

Chi-square 
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The  above  questions  require  several  statistical 
techniques  to  provide  a  confident  analysis.  Since 
parametric  tests  were  used,  the  assvimption  of  normality  was 
required.  All  variables  which  contained  a  Nilk-Shapiro 
statistic  for  goodness-of-f it  to  the  normal  distribution  in 
excess  of  .70  were  considered  to  meet  the  normality 
requirement  (Shane,  1991). 

T-test .  The  t-test  involves  the  comparison  of  two  sets 
of  sample  data  to  draw  conclusions  regarding  the  association 
of  their  underlying  distributions.  The  two  types  of  t-test 
that  are  used  in  this  research  are  the  independent -sample 
and  paired  t-tests. 

An  independent -sample  t-test  can  be  performed  to 
determine  if  the  sample  data  was  probably  drawn  from 
different  distributions.  Assumptions  regarding  this 
application  are  that  both  populations  that  provided  the 
samples  are  normally  distributed  and  the  population 
variances  are  equal .  Since  the  estimate  of  the  variance  is 
drawn  from  both  sample  data  sets,  often  a  pooled  estimator 
is  used  to  identify  the  coiranon  variance.  This  pooled 
estimator  provides  a  weighting  to  the  variance  that 
considers  the  relative  sise  of  each  sample  (Devore, 
1987;334-336). 

The  sample  means  and  common  variance  are  used  to 
compute  a  t-statistic  for  a  two  sided  test.  High  t- 


62 


statistics  translate  a  lower  probability  that  the  samples 
are  from  the  same  distribution. 

While  the  normal  t-test  uses  independent  samples,  the 
paired  t-test  involves  the  comparison  of  two  separate  but 
dependent  samples.  In  this  research,  many  of  the 
comparisons  involve  pre/post  test  data  where  the  measuring 
instrument  and  test  subject  are  the  same  and  the  effect  of 
treatment  (training)  is  what  is  being  measured.  In  this 
case,  the  two  sets  of  data  are  dependent  and  the  t  statistic 
developed  provides  insight  as  to  whether  the  treatment  had  a 
significant  effect  on  the  subject. 

The  statistic  of  interest  is  the  difference  between  the 
mean  values  for  each  measurement.  The  pre/post  test 
approach  is  designed  to  remove  the  effects  of  factors  other 
than  the  treatment  to  determine  a  measurable  difference 
(Hamburg,  1977:306).  No  change  due  to  the  treatment  would 
mean  that  there  was  no  statistically  discernable  difference 
in  the  paired  observations  (i.e.  the  treatment  had  no  impact 
on  what  was  measured).  The  assumptions  underlying  the 
paired  t-test  are  that  the  selected  pairs  are  independent 
and  the  difference  in  pairs  are  normally  distributed.  A 
common  variance  is  not  required  in  this  case  since  the 
variable  of  interest  is  the  difference  in  sample  means,  and 
this  variable  has  its  own  variance  that  can  estimate  the 
population  variance  (Devore,  1987;344-345) . 
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Regression.  Regression  analysis  involves  determining 


the  statistical  relationship  between  two  or  more  variables 
(Devore,  1987:451).  In  simple  linear  regression  analysis, 
values  of  independent  variables  are  often  helpful  in 
predicting  the  value  of  a  dependent  variable.  In  CAT  leader 
training  specifically,  investigative  research  question 
number  3  involves  the  ability  of  three  variables  to  predict 
the  value  of  the  subject's  knowledge  pre-test  score.  Those 
three  predictors  are  past  training,  past  education,  and  past 
experience. 

The  equation  of  interest  is  linear  and  is  defined  as: 

T«  83X3+ e  .  Y  is  the  dependant  variable  and 

each  portion  of  the  right  hand  side  of  the  equation  defines 
the  corresponding  slope  (8)  associated  with  the  three 
independent  variables.  The  e  represents  the  error  term  of 
the  model  and  is  assumed  to  be  normally  distributed  with  a 
mean  of  xero.  Each  of  the  independent  variables  is 
considered  a  significant  predictor  of  the  knowledge  test,  if 
its  slope  is  greater  than  zero. 

It  is  important  to  examine  the  aptness  of  the  model  for 
the  data  being  used  before  extensive  analysis  is  performed 
(Neter,  et  al . ,  1990:113).  Aptness  provides  confidence  that 
the  regression  model  is  competent  for  use  (Reynolds,  1991). 

Study  of  the  residuals  is  one  method  to  ensure  that  the 
model  assximptions  fit  the  data  being  examined.  Plots  of 
residuals  against  independent  variables,  fitted  values. 
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time,  and  omitted  independent  variables  are  methods  to 
visually  check  data  trends  that  result  in  model  departures. 
(Neter,  et  al . ,  1990:117).  Informal  diagnostic  review  of 
residual  plots  provide  information  on  whether  departures 
from  the  simple  linear  regression  model  would  preclude  its 
use. 

Chi-square  test.  The  chi-square  (x^)  distribution  is  a 
probability  distribution  that  is  defined  by  the  parameter  v 
(degrees  of  freedom).  Often  the  chi-square  distribution  is 
used  to  identify  the  goodness  of  fit  between  observed 
frequencies  and  theoretical  frequencies  (Hamburg,  1977:318). 
In  some  of  the  investigative  questions  considered,  the  issue 
of  whether  individuals  meeting  specific  criteria  are 
represented  by  categories  that  are  randomly  distributed. 

In  this  research,  each  category  of  a  variable  being 
investigated  is  considered  to  be  mutually  exclusive, 
collectively  exhaustive,  and  randomly  distributed.  If  this 
is  not  the  case,  then  some  categories  of  a  variable  will 
have  a  disproportionate  level  of  subjects.  The  chi-square 
goodness  of  fit  test  is  a  method  that  allows  you  to  test  if 
a  disproportionate  level  exists.  The  chi-square  test 
requires  that  each  category  contain  at  least  five 
observations . 

These  three  testing  techniques  defined  above  represent 
the  statistical  approach  used  in  drawing  conclusions  to  the 
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investigative  questions  from  the  research  data.  These 
results  will  be  addressed  in  the  next  chapter. 

Summary 

This  chapter  outlined  the  methodology  associated  with 
accomplishing  the  research  objectives  and  answering  the 
specific  research  questions.  The  development  and  validation 
of  a  set  of  behavioral  learning  objectives  provided  the 
framework  for  both  an  internal  and  field  evaluation  of  the 
training  through  the  use  of  a  knowledge  test  and 
questionnaire,  respectively. 

With  Chapters  1  through  4  as  a  background  and  the  data 
collected  by  the  knowledge  test  and  questionnaire,  results 
and  recommendations  can  now  be  addressed  and  discussed. 
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V.  Presentation  and  Discussion  of  Results 


Introduction 

As  stated  in  Chapter  1,  the  investigative  questions 
provided  the  specific  guidelines  for  the  research  and  were  a 
means  for  a  detailed  breakdown  of  the  research  objectives. 
Consequently,  they  are  the  primary  topics  of  discussion 
regarding  results. 

Each  investigative  question  will  now  be  identified  and 
discussed  based  on  the  data  collected  and  statistical  tests 
performed. 

Investigative  Question  1 

Is  the  basic  curriculum  described  by  the  information  in 

the  CAT  leader  notebook  taught  by  all  course 

facilitators? 

Chapter  4  discussed  the  development  of  learning 
objectives  both  to  measure  the  three  course  terminal 
objectives  and  to  provide  a  basis  for  development  of  a 
knowledge  test  and  questionnaire.  The  information  to 
develop  the  learning  objectives  came  from  the  CAT  leader 
notebook  which  contained  the  course  syllabus  and  related 
material.  Nine  of  the  12  instructors  responded  to  the  list 
of  proposed  objectives  located  in  Appendix  B  and  provided 
the  data  for  content  validity  ratio  (CVR)  calculations.  The 
values  of  individual  CVRs  for  objectives  is  located  in 
Appendix  C. 

Table  3  identifies  that  of  the  62  objectives  proposed, 
55  had  a  positive  CVR  which  indicates  that  the  majority  of 
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instructors  completely  taught  those  objective  when 
responsible  for  that  training  block  during  class.  The  7 
items  with  a  negative  CVR  were  not  considered  valid 
objectives  taught  by  instructors  since  less  than  half  stated 
they  completely  covered  the  objective  in  the  course.  These 
seven  included: 

Objective  4:  Identify  the  expected  life  of  the  four 
major  sub-systems  that  make  up  an  organizations'  TQ 
structure  from  a  list  of  options. 

Objective  6:  Have  an  appreciation  for  the  difference 
between  maintenance  factors  and  motivational  factors 
identified  by  Herzberg. 

Objective  23:  Have  an  appreciation  for  the  strategies 
of  managing  difficult  people  developed  by  Manning  and 
Haddock. 

Objective  24:  Identify  some  barriers  to  effective 
communication. 

Objective  26:  Know  the  principle  for  giving  and 
receiving  feedback. 

Objective  57:  Identify  a  diagram  in  the  CAT  leader 
notebook  which  represents  a  Solution  Selection  Matrix. 

Objective  60:  Identify  a  diagram  in  the  CAT  leader 
notebook  which  represents  a  Force  Field  Diagram. 

The  content  validity  index  (CVI)  which  is  the 

arithmetic  mean  for  all  positive  CVR  items,  had  a  value  of 

.73.  It  represents  the  extent  of  perceived  overlap  existing 

between  the  objectives  identified  and  the  material  taught  in 

the  course.  This  high  value  indicates  that  there  is  a 

substantial  overlap  between  the  objectives  and  the  course 

content . 

Results  indicate  fifty-five  of  the  62  objectives  were 
completely  taught  by  the  majority  of  instructors  with  36  of 


Table  3 

Results  of  Content  Validity  Calculations 


Objectives  covered  by  all  instructors  (p  <  .05)  36 

Objectives  covered  by  majority  of  instructors  19 

Objectives  not  covered  by  majority  of  instructors  _ 7 

Total  62 


Content  Validity  Index  of  top  55  objectives  =  0.73 


those  covered  by  all  instructors  (p  <  .05).  This  indicates 
that  the  instructors  rated  the  content  of  their 
presentations  as  sufficient  to  cover  the  fifty-five 
objectives.  Whether  the  student  actually  did  perform  to  a 
level  required  by  the  objectives  is  a  topic  for 
investigative  question  2. 

It  should  be  stressed  that  this  was  a  self-rating  scale 
based  on  the  material  as  presented  by  an  instructor.  Some 
concern  about  self-rating  is  warranted,  but  not  totally 
supported  by  the  data.  First,  a  wide  range  of  CVRs  was 
noted,  indicating  that  instructors  did  not  provide  the 
maximum  rating  of  1  in  all  cases,  and  did  identify  where 
objectives  were  not  completely  covered.  Second,  the  rating 
list  in  Appendix  B  was  provided  to  the  facilitators  as  a  set 
of  "proposed"  objectives.  "Proposed"  carries  less  of  an 
evaluation  stigma  which  may  impact  the  instructors'  scoring. 
If  instructors  perceived  their  review  of  objectives  as  an 
evaluation  tool,  they  would  be  more  likely  inflate  their 
scores.  Requesting  anonymous  responses  to  a  list  of 
"proposed"  objectives  attempted  to  limit  this  problem. 
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While  three  of  the  part-time  facilitators  did  not 
respond  to  follow-up,  these  results  do  provide  confidence 
that  these  objectives  properly  reflect  course  material  since 
the  respondents  who  did  respond  represent  over  85%  of  the 
instructor  cadre  and  teach  above  that  proportion  of  the 
total  course. 

Conclusion .  The  CVR  and  CVI  scores  indicate  that  55  of 
the  62  proposed  objectives  developed  from  the  material  in 
the  CAT  leader  notebook  are  presented  by  the  facilitators 
during  the  four  day  course.  They  represent  an  initial  set 
of  validated  objectives  and  are  useful  in  developing  the 
knowledge  test  and  questionnaire.  The  seven  objectives  not 
taught  completely  by  instructors  are  not  necessarily 
improper  for  the  course,  but  do  not  receive  the  same 
emphasis  as  other  material.  As  a  result,  test  items 
developed  from  these  seven  objectives  will  not  be  considered 
in  the  analysis  of  overall  knowledge  test  results. 

Investigative  Question  2 

Does  CAT  leader  training  improve  the  knowledge  of 

participants  in  each  of  the  training  blocks  presented? 

The  validated  objectives  discussed  in  investigative 
question  1  provide  a  basis  for  determining  if  knowledge 
increased  as  a  result  of  the  course.  Of  the  55  objectives, 
27  required  more  than  just  an  appreciation  for  a  concept. 
These  objectives  provided  the  basis  for  the  knowledge  test 
and  began  with  the  word  "identify”  or  "know". 
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The  knowledge  test  scoring  results  are  divided  into  two 
areas:  topics  related  to  training  blocks  and  individual 
tools  and  techniques. 

Table  4  identifies  the  overall  results  from  the  three 
classes  (N=44)  evaluated  by  the  knowledge  test.  Total  score 
results  are  identified  in  the  first  row  and  then  material  is 
broken  down  by  topics  tested.  The  last  four  entries 
describe  methods  of  testing  tools  and  techniques.  The 
number  of  questions,  mean  score  change  from  the  pre-test  to 
the  post-test,  and  paired  t-test  results  are  provided  in 
columns  two  to  four.  A  t-test  result  lower  than  0.05  in  row 
number  four  indicates  a  95%  confidence  that  a  significant 
improvement  is  made  in  the  post-test  score. 

All  variables  associated  with  valid  objectives 
(positive  CVR)  were  checked  for  normality  using  the  Wilk- 
Shapiro  statistic.  Since  the  variables  measured  the 
improvement  in  score  from  the  pre-test  to  the  post-test,  a 
discrete  distribution  resulted.  Even  though  the  normal 
distribution  is  continuous,  the  range  of  discrete  responses 
allowed  comparison  to  a  normal  distribution. 

In  all  cases  but  two,  variables  were  found  to  very 
o'*.  u.9'ly  approximate  a  normal  distribution  (Wi Ik-Shapiro 
statistic  >  .90).  The  two  variables  that  had  distributions 
that  less  closely  resembled  the  normal  distributions 
involved  the  Strength  Deployment  Inventory  (SDI)  (Wilk- 
Shapiro=.68}  and  Leadership  (Wilk-Shapiro= . 71) .  Each  of 
these  variables  were  represented  by  one  question  on  the 
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knowledge  test  which  limited  the  range  of  values  in  their 
discrete  distributions.  Since  Wilk-Shapiro  values  in  excess 
of  .70  were  considered  acceptable  in  assuming  normality,  all 
topics,  except  SDI,  met  the  established  normality  criteria 
for  parametric  tests  (Shane,  1991). 


Table  4 

Training  Block  Results  -  Overall 


Training  Block  No.  of  Mean  Prob>|T| 

Questions  Change 

Total  Score 

56 

9.90 

0.00 

Total  Quality 

12 

1.61 

0.00 

SDI 

1 

0.14 

** 

Group  Stages 

10 

1.39 

0.00 

Leadership 

1 

0.07 

0.41 

Problem  Solving  Steps 

6 

1.36 

0.00 

Types  of  Data 

4 

0.36 

0.14 

Definitions 

9 

2.27 

0.00 

Use  of  Tools 

4 

0.39 

0.07 

Steps  of  a  tool 

5 

1.07 

0.00 

Select  diagrams 

4 

1.25 

0.00 

**  =  did  not  meet  normality  assumption  for  t-test 

Overall  Score  Improvement.  The  mean  change  in  overall 
score  fur  the  three  classes  was  9.9  for  the  56  question 
test.  This  shows  that  the  average  student  improved  their 
correct  response  score  by  almost  10  questions  from  the  pre¬ 
test  to  the  post-test. 
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The  concern  about  test  sensitizing  due  to  the  short 
time  between  pre-test  and  post-test  is  addressed  by  the 
post-test  only  group.  The  knowledge  test  design  was  based 
on  a  pre-test/post-test  pre-experimental  design  involving  a 
control  group.  This  control  group  was  not  exposed  to  the 
knowledge  test  until  after  the  course  was  completed.  Their 
post-test  results  (N=17)  were  compared  to  the  combined  post¬ 
test  results  (N=44)  from  the  other  classes  and  a  pooled  t- 
test  was  performed  on  the  overall  test  score.  The  pooled  t- 
test  involves  two  distributions  assumed  to  be  normally 
distributed  and  uses  a  pooled  estimator  of  the  common 
variance.  (Devore,  1987:334-336). 

The  resulting  t-statistic  produced  a  p-value  equal  to 
.07  and  does  not  indicate  that  a  significant  improvement  in 
score  is  attributable  to  exposure  to  the  knowledge  test 
prior  to  the  course.  The  positive  t-value  does  indicate  a 
slightly  improved  post-test  score  from  the  control  group, 
but  not  one  large  enough  to  draw  a  separate  conclusion 
(p  <  .05). 

Individual  Training  Material  Improvement.  The  results 
in  the  total  score  row  of  Table  4  involve  all  the  test 
questions  that  make  up  the  overall  score.  All  training 
blocks  are  represented  except  Block  6  (meeting 
effectiveness)  which  had  no  objectives  measurable  by  pencil 
and  paper  testing.  Of  the  nine  sets  of  questions  identified 
that  met  the  normality  assumption,  six  showed  significant 
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improvement  and  three  showed  no  significant  improvement 
(P  <  .05).  SDI  t-test  results  were  not  computed. 

The  six  showing  improvement  questions  relating  to: 
total  quality,  Tuckman's  stages  of  group  development,  steps 
in  the  problem  solving  process,  tools  and  techniques 
definitions,  the  steps  of  specific  tools  and  techniques,  and 
diagrams  associated  with  specific  tools  and  techniques. 

These  six  sets  represented  52  of  the  sixty-three  questions 
in  the  knowledge  test. 

The  three  sets  of  questions  that  did  not  indicate  a 
significant  improvement  consisted  of  nine  of  the  10 
remaining  questions.  They  included  the  subjects  of: 
functions  of  a  leader,  types  of  data,  and  uses  for  specific 
tools  and  techniques. 

The  types  of  data  test  questions  involved 
differentiating  between  qualitative  and  quantitative  data. 
The  knowledge  test  had  four  questions  addressing  types  of 
data.  Two  questions  required  students  to  identify  problem 
solving  tools  and  techniques  associated  with  either 
qualitative  or  quantitative  data.  Two  additional  questions 
required  students  to  differentiate  between  the  types  of  data 
from  a  list  of  examples.  Since  the  scores  did  not  improve 
significantly  (p  <  .05),  students  appeared  to  have  a  limited 
improvement  in  understanding  the  definitions  and 
applications  of  data  types. 

Also,  use  of  specific  tools  and  techniques  did  not  show 
improvement.  Four  test  items  required  the  student  to 
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identify  from  a  list  the  inappropriate  tool  or  technique 
associated  with  a  problem  solving  step  (defining  a  problem, 
gathering  information,  etc.).  The  appropriateness  of 
specific  tools  and  techniques  was  defined  by  a  summary  sheet 
provided  in  the  CAT  leader  notebook.  Since  scores  did  not 
improve  significantly  (p  <  .05),  students  appear  to  have  a 
limited  improvement  in  understanding  the  appropriate  use  of 
tools  and  techniques  in  specific  problem  solving  steps. 

Review  of  both  sets  of  test  item  scores  indicates  that 
lack  of  improvement  was  not  due  to  relatively  high  pre-test 
scores  which  would  have  left  little  room  for  improvement. 

Since  only  one  question  each  in  the  test  involved  SDI 
and  leadership,  their  lack  of  improvement  may  partially  be 
explained  by  the  small  number  of  questions  which  limited  the 
range  of  scores  for  each  subject.  Limiting  this  range 
results  in  overall  smaller  variation  among  scores,  making  it 
more  difficult  to  identify  significant  changes. 

Another  method  to  identify  significant  changes  (or  lack 
of)  in  the  testing  group  is  to  review  the  results  by 
individual  classes. 

Individual  Class  Results.  Individual  class  scores  were 
combined  to  determine  the  overall  results.  Since  each  class 
had  its  own  population  and  teaching  environment,  their 
individual  results  show  several  aspects  of  training  impacts 
not  visible  in  the  overall  scores.  Table  5  breaks  down  the 
training  material  results  per  class. 
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Table  5 

Training  Block  Results  -  Per  Class 


Improved  Score 

Training  Blocks  (p  <  *05} 

April  May  June 

Total  Score* 

X 

X 

X 

Total  Quality* 

X 

X 

X 

Oroup  Stages* 

X 

X 

Leadership 

Prob  Solving  Steps* 

X 

X 

X 

Types  of  Data 

X 

Definitions* 

X 

X 

X 

Use  of  Tools 

X 

Steps  of  a  tool* 

X 

X 

Select  diagrams* 

X 

X 

X 

X  •  Improvement  *  ^  Overall  Improvement  (p  <  .05)  | 

In  every  case  where  an  overall  improvement  was  noted, 
at  least  two  of  the  three  individual  classes  showed  a 
significant  improvement.  This  would  indicate  that  no 
specific  class  significantly  biased  overall  results  by 
extreme  scores. 

Several  factors  were  considered  to  explain  the  lack  of 
in^rovement  in  areas  by  all  classes.  Student  apathy  toward 
taking  the  post-test  was  considered  as  a  possible  reason  for 
the  lower  scores;  however,  improved  scores  in  later  test 
sections  did  not  support  that  assertion.  The  lack  of 
improvement  was  not  due  to  different  instructors  since  the 
sasie  instructors  were  used  for  each  course  offering. 
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Discussions  with  one  of  the  course  facilitators 
indicated  no  departures  from  teaching  methods  for  any  of  the 
classes.  However,  one  unique  aspect  of  the  June  class 
involved  the  training  environment.  Both  the  May  and  June 
classes  were  held  in  a  renovated  school  building  that  used 
loud  and  less  efficient  window  air  conditioners.  For  the 
May  class,  the  distractions  were  minimised  by  cooler 
weather.  However,  for  the  June  class,  the  facilitator 
identified  these  window  units  as  distracting  during 
presentations  and  resulting  in  a  warmer  training 
environment.  The  results  of  the  June  class  scores  may  be 
partially  a  result  of  this  environment,  but  this  possibility 
cannot  be  substantiated  with  the  data  provided. 

Overall,  since  there  was  no  significant  change  in  score 
(p  <  .05),  the  lack  of  improvement  should  not  be  contributed 
(necessarily)  to  teaching  methods,  testing  methods,  or 
enviroiunent .  Howevei .  all  of  these  issues  should  be 
investigated  when  considering  a  means  to  increase  and 
measure  learning  in  these  areas. 

Of  the  thirty  improvements  possible  (all  three  classes 
in  each  of  ten  questions  sets)  for  those  meeting  the 
normality  criteria,  twenty-one  improvements  were  noted. 
Twelve  of  these  improvements  occurred  in  the  section 
covering  the  problem  solving  tools  and  techniques  presented 
in  the  course.  Since  they  represent  a  large  portion  of  both 
the  course  and  test  material,  their  results  are  broken  out 
separately. 


Tools  and  Techniques  Results.  Twenty-two  of  the  56 


questions  involved  the  use  of  problem  solving  tools  and 
techniques.  To  provide  insight  into  the  learning  of  this 
specific  material,  the  associated  questions  were  re-scored 
by  the  type  of  tool  or  technique.  The  overall  and 
individual  class  results  are  provided  in  Tables  6  and  7, 
respectively. 


Table  6 

Tools  and  Techniques  Results  --  Overall 


Tool  or  Technique 

Ho.  of 
Questions 

Mean 

Score 

Prob> IT! 

Total  Score 

22 

4.98 

0.00 

Brainstorming 

2 

0.04 

0.56 

Nominal  Group  Tech 

2 

0.34 

0.01 

Cause  and  Effect 

2 

0.95 

0.00 

Pareto  Analysis 

3 

1.16 

0.00 

Check  Sheets 

4 

0.91 

0.00 

Flow  Charting 

2 

0.09 

0.35 

Solution  Sel  Matrix 

2 

0.32 

0.04 

Force  Field  Diagrams 

3 

1.20 

0.00 

Action  Plans 

2 

0.39 

0.00 

Wilk-Shapiro  statistics  were  computed  for  these 
variables  to  ensure  the  normality  assumption  was  proper. 
Values  in  excess  of  .70  were  noted  for  all  variables  making 
the  use  of  a  paired  t-test  valid  in  all  cases. 

The  total  score  was  provided  by  combining  the  twenty- 
two  associated  tools  and  techniques  questions.  Table  6 
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Table  7 

Tools  and  Technique  Results  -  Per  Class 


Tool  or  Technique  Improved  Score 

(P<.05) 

Apr  May  Jun 

Total  Score* 

X 

X 

X 

Brainstorming 

Nominal  Group  Tech* 

X 

Cause  and  Effect* 

X 

X 

X 

Pareto  Analysis* 

X 

X 

X 

Check  Sheets* 

X 

Plow  Charting 

Solution  Sel  Matrix* 

X 

Force  Field  Diagrams* 

X 

X 

Action  Plans* 

X 

i  X  s  Improvement  *  -  Overall  Improvement  (p  <  .05)  | 

shows  that  the  average  improvement  from  the  pre*test  to 
post-test  was  4.98  questions  per  student  (23%).  This 
indicates  a  significant  improvement  in  scores  relating  to 
this  material  (p  <  .05).  Of  the  ten  tools  and  techniques 
involved,  only  two,  brainstorming  and  flow  charting,  did  not 
show  a  significant  improvement  after  training. 

Brainstorming  and  flow  charting  also  represented  the 
only  two  tools  or  techniques  where  none  of  the  three  classes 
showed  any  significant  improvement,  ffhile  the  lack  of 
overall  ii^rovenient  for  training  material  described  earlier 
(Types  of  Data  and  Use  of  Tools)  is  partially  explained  by 
the  decrease  in  scoring  by  individual  classes,  this 
condition  is  not  apparent  here.  However,  it  is  noteworthy 
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that  brainstorming  and  flow  charting  both  represent  the 
highest  pre->test  scores  for  any  tool  or  technique  as  shown 
in  Table  8. 


Table  8 

Tools  and  Techniques  Ranked  Pre-test  Results 


Tool  or  Technique 

Pre-test 

score 

Mo.  of 
Questions 

Avg 

Brainstorming 

1.68 

2 

84% 

Plow  Charting 

1.50 

2 

75% 

Action  Planning 

1.20 

2 

60% 

Nominal  Group  Tech 

1.18 

2 

59% 

Solution  Sel  Matrix 

0.93 

2 

47% 

Pareto  Analysis 

1.23 

3 

41% 

Check  Sheets 

1.45 

4 

36% 

Force  Field  Diagrams 

0.95 

3 

32% 

Cause  and  Effect 

0.57 

2 

29% 

This  means  that  a  very  large  increase  was  needed  in  an 
already  good  score  to  show  a  statistically  significant 
improvement.  Since  two  questions  were  associated  with 
brainstorming  and  flow  charting,  the  concern  over  a  limited 
number  of  categories  in  the  discrete  distribution  is  less  of 
an  issue  here  than  with  the  variables  for  SDl  and 
leadership.  The  limited  number  of  categories  does  pose  a 
slight  problem,  but  it  seems  it  is  compounded  by  the  high 
pre-test  scores  for  both  variables. 

The  June  class  showed  the  smallest  improvement  (3  out 
of  10  topics)  of  any  of  the  three  classes.  As  described 
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•arlier,  m  noisy  and  less  comfortable  classroom  environment 
was  present  and  may  have  contributed  to  the  lower  scores. 

Conclusion.  Tables  5  and  7  provide  good  suiranary  charts 
for  training  material  and  tools  and  techniques  performance 
results.  Those  areas  where  improvement  is  statistically 
significant  represent  topics  where  knowledge  appears  to  be 
increased  by  training.  Areas  where  no  significant 
improvement  is  noted  represent  topics  that  facilitators  can 
focus  on  to  improve  course  effectiveness.  No  conclusions 
were  drawn  from  the  data  concerning  SDI,  since  it  failed  to 
meet  the  criteria  for  assuming  a  normal  distribution. 

The  ranked  pre-test  results  provided  in  Table  8 
indicated  that  the  highest  level  of  competence  for  tools  and 
techniques  prior  to  training  were  in  brainstorming  and  flow 
charting.  Students  did  very  well  initially  in  these  two 
areas  and  did  much  worse  in  other  areas  that  may  need  more 
emphasis  in  the  future. 

Investigative  Question  3 

Are  the  following  antecedent  variables  significant 
predictors  of  a  student's  performance  in  the  knowledge 
pre-test  (Y)? 

XI.  Involvement  in  previous  CATs  (Experience) 

X2.  Previous  problem  solving/team  dynamics 
training  (Training) 

X3.  Education  Level  (Education) 

A  general  linear  regression  model  was  developed  from 

these  three  predictors  involving  training,  experience,  and 

education.  Experience  ranged  from  no  involvement  to 

involvement  in  three  or  more  CATs.  The  training  predictor 
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was  based  on  how  the  participant  answered  5  items  relating 
to  past  training  that  addressed  current  course  material . 
Education  level  was  scaled  on  the  highest  degree  received  to 
date  ranging  from  high  school  diploma  to  a  doctoral  degree. 

To  measure  predictive  capability,  the  all^ways 
regression  procedure  was  used  to  allow  variables  with  a  0.05 
significance  level  entry  into  the  model. 

Model  aptness  was  tested  by  several  factors  that  were 
identified  in  the  regression  model  discussion  in  Chapter  4. 
Testing  of  residuals  indicated  a  normal  random  distribution 
(Wilk-Shapiro  =  0.95)  with  a  mean  of  zero  and  no  trends  with 
respect  to  predictors.  As  shown  in  Table  9, 
multicol linearity  was  not  a  problem  since  none  of  the 
predictor  variables  exhibited  a  very  high  correlation  with 
other  variables. 


Table  9 

Correlation  of  Pre-test  Predictor  Variables 


Variable 

Degree 

Training 

CAT  Involvement 

Degree 

1 

.413 

.278 

Training 

.413 

1 

.113 

CAT  Involvement 

.278 

.113 

1 

Since  aptness  tests  regarding  residuals  indicated  no 
deviation  from  model  requirements,  the  regression  analysis 
using  the  pre-test  score  as  the  dependant  variable  was 
performed.  Results  are  shown  in  Table  10  and  indicate  that 
the  independent  variables  for  education  (degree)  and 
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previous  training  (training)  were  significant  predictors 
while  previous  CAT  involvement  (experience)  was  not  a 
significant  predictor  (P  <  .05). 


Table  10 

Svuranary  of  the  Regressior  Model 


Partial  R* 

Model  R^ 

Prob>F 

Degree 

0.09 

0.09 

0.04 

Training 

0.11 

0.20 

0.02 

Conclusion.  While  the  variables  for  training  and 
degree  are  significant  predictors,  the  overall  value  for 
the  model  including  these  two  variables  is  .20,  indicating 
it  can  only  account  for  20%  of  all  the  variance  found  in  the 
scores.  Previous  CAT  involvement  is  not  a  significant 
predictor.  As  a  result,  past  training  and  education  may  be 
useful  in  a  more  comprehensive  model,  but  alone  provide 
limited  explanatory  power  in  predicting  an  individuals  pre¬ 
test  score. 

Investigative  Question  4 

Is  there  an  overall  increase  in  self-efficacy  in  the 
application  of  course  material  as  a  result  of  attending 
CAT  leader  training? 

As  part  of  both  the  pre-test  and  post-test,  each 
student  was  asked  to  rate  their  ability  to  perform  various 
tasks  related  to  the  application  of  CAT  leader  training 
material.  These  tasks  included  having  the  capability  to: 
lead  a  group  through  the  problem  solving  process,  explain 
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the  TQ  structure  and  philosophy,  use  the  tools  and 
techniques  of  problem  solving,  prepare  and  facilitate  an 
effective  meeting,  and  understand  the  subject  of  group 
dynamics  enough  to  build  an  effective  team. 

Since  all  five  relate  to  training  material,  the  scores 
from  a  seven-point  Likert  scale  of  each  were  combined  to 
provide  an  overall  self-efficacy  rating.  Wilk-Shapiro 
statistics  on  the  related  variables  indicated  approximately 
normal  distributions  which  allowed  the  use  of  the  paired  t- 
test  for  comparison.  These  scores  are  provided  in  Table  11. 


Table  11 

Self  Efficacy  Results  -  Per  Class 


Group 

Pre-test 

Score 

Chanoe 

Prob>lTl 

Improved 

Overal 1 

17.5 

5.72 

0.00 

X 

April 

17.8 

6.19 

0.00 

X 

May 

17.0 

7.50 

0.00 

X 

June 

18.0 

2.54 

0.45 

An  increase  in  self-efficacy  relating  to  the  training 
material  is  observed  across  all  three  classes.  A  paired  t- 
test  shows  that  the  overall  score  and  the  classes  tested  in 
April  and  May  increased  their  self-efficacy  score 
significantly.  The  June  class  showed  less  than  a 
significant  improvement  in  this  area. 

Comparison  of  the  pre-test  scores  rules  out  the 
possibility  that  the  June  class  had  a  significantly  higher 
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score  which  left  little  room  for  improvement.  On  the 
contrary,  the  June  class  had  only  a  slightly  higher  score 
but  showed  a  significantly  lower  improvement  in  the  post¬ 
test  . 

This  finding  also  correlates  with  the  absolute  decrease 
in  scoring  for  test  items  involving  types  of  data  and  use  of 
tools.  As  discussed  earlier  in  the  overall  results  broken 
down  by  class,  the  Jtune  class*  warmer  and  noisier  training 
environment  was  the  only  area  specifically  identified  by  one 
of  the  course  facilitators  that  seemed  different  from  the 
other  two  classes.  However,  since  this  class  did  exhibit 
some  improved  scores  in  certain  areas,  environment  does  not 
seem  to  have  had  a  detrimental  impact  in  all  data 
measurements.  As  a  result,  the  training  environment  can  be 
considered  as  a  possible  influence  on  test  scores  and  self- 
efficacy  ratings,  but  data  does  not  support  it  as  the  only 
factor  that  may  explain  the  variance  between  classes. 

Individual  Subject  Self-Efficacy  Scores.  The  five 
tasks  that  were  used  to  provide  the  overall  self-efficacy 
score  were  broken  out  to  investigate  any  trends  in  this  data 
for  the  combined  group  of  three  classes.  Normality 
requirements  were  met  in  these  areas  which  allowed  paired  t- 
test  to  be  performed.  Table  12  provides  the  self-efficacy 
scores  by  task. 

Results  indicated  that  in  four  of  the  5  individual 
tasks,  the  overall  group  showed  a  significant  improvement. 
The  remaining  task  (leading  a  group  through  the  problem 
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Table  12 

Individual  Task  Self-efficacy  Results  -  Overall 


Self-ef f icacv  Subiect 

Pre-test 

Mean 

Prob>lT! 

Mean 

Change 

Overal 1 

17.54 

5.72 

0.00 

Lead  a  group 

4.34 

0.45 

0.06 

Explain  TQ 

3.11 

1.34 

0.00 

Use  tools  and  techniques 

3.50 

1.02 

0.00 

Run  an  effective  meeting 

3.50 

1.11 

0.00 

Build  an  effective  team 

3.07 

1.60 

0.00 

solving  process),  while  not  showing  a  significant 
improvement,  did  have  the  highest  pre-test  score. 

Conclusion.  Overall  improvement  in  self-efficacy  for 
two  of  the  3  classes  was  large  enough  to  indicate  a 
significant  improvement  for  the  entire  group.  Investigation 
of  individual  task  self-efficacy  showed  an  improvement  in 
four  of  the  five  task  scores  showing  that  a  large  portion  of 
the  course  improves  the  students'  perception  of  their 
capability  to  perform  specific  tasks. 

Investigative  Question  5 

Are  a  significant  number  of  CAT  leaders  assigned  CAT 

issues  within  three  months  of  training? 

Investigative  question  five  begins  the  field  evaluation 
portion  of  the  research  by  looking  at  data  relating  to  the 
past  course  participants.  Of  the  208  responses  to  the 
questionnaire,  56  past  attendees  indicated  that  they  have 
served  as  a  CAT  leader  since  training.  These  individuals 
are  the  focus  of  this  investigative  question. 
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To  determine  whether  the  majority  of  CAT  leaders  are 
assigned  a  CAT  issues  soon  after  training,  those  who 
indicated  they  have  served  as  a  CAT  leader  since  training 
were  asked  to  identify  the  time  gap  between  training  and 
being  assigned  a  CAT  issue.  Table  13  identifies  the 
frequency  of  responses  by  those  surveyed. 

Table  13 

Gap  Between  Training  and  CAT  Leader  Assigximent 


CAT  assigned  prior  to  training  12 
Cat  assigned  0-1  months  after  training  12 
CAT  assigned  2-3  months  after  training  14 
CAT  assigned  4-6  months  after  training  13 
CAT  assigned  7-12  months  after  training  4 
CAT  assigned  over  1  year  after  training _ 1 


A  chi-square  test  was  preformed  to  determine  if  the 
responses  followed  a  homogeneous  distribution.  Three  groups 
were  formed  by  combining  the  first  three  and  the  last  two 
categories  to  test  whether  a  significant  number  of  CAT 
leaders  were  assigned  issues  within  three  months  of 
training.  Combining  the  first  three  categories  was  required 
because  they  represented  the  time  period  of  less  than  three 
months.  The  last  two  categories  were  combined  to  meet  the 
requirement  of  having  at  least  five  responses  in  each 
category  to  obtain  a  valid  chi-square  statistic. 

Results  yielded  a  chi-square  statistic  of  7.14  which 
was  sufficiently  large  to  reject  the  homogeneous 
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distribution  test  (P  <  .05).  This  indicated  that,  of  those 
receiving  CAT  assignments,  a  significant  number  of  CAT 
leaders  were  assigned  CATs  within  three  months  of  training. 

Further  review  of  the  data  showed  almost  a  homogeneous 
distribution  over  the  first  four  categories.  To  determine 
if  the  training  gap  supported  this  observation,  another  chi- 
square  test  was  performed.  This  time  the  last  three 
categories  were  combined  to  represent  those  leaders  who  were 
assigned  a  CAT  four  or  more  months  after  training.  Results 
yielded  a  chi-square  statistic  of  1.72  which  was 
sufficiently  small  enough  not  to  reject  the  homogeneous 
distribution  hypothesis  (p  <  .05). 

Conclusion.  A  significant  number  of  those  training 
participants  who  become  CAT  leaders  are  assigned  a  CAT  issue 
within  three  months  of  training.  Additionally,  the 
underlying  distribution  describing  the  time  interval  of 
assignments  follows  a  homogeneous  distribution  where  one 
quarter  of  the  assignments  are  made  in  each  of  the 
following  categories:  before  training,  within  a  month  of 
training,  between  two  and  three  months  after  training,  and 
over  four  months  after  training. 

Investigative  Question  6 

Are  there  a  significant  nvunber  of  training  participants 

becoming  members  of  CATs? 

CAT  leader  training  was  designed  to  support  the  total 
quality  initiatives  in  Aeronautical  Systems  Division. 
However,  the  concepts,  tools  and  techniques  of  the  training 
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are  useful  in  many  organizational  situations.  This  has 
encouraged  the  attendance  of  individuals  who  do  not  plan  on 
being  a  CAT  leader,  but  wish  to  improve  their  problem 
solving  skills.  Investigative  question  6  addresses  the 
issue  of  CAT  participation  by  past  course  participants. 

Of  the  208  respondents  to  the  questionnaire,  116  (57%) 
indicated  that  they  have  not  been  involved  as  either  a 
leader  or  participant  in  a  corrective  action  team.  Of  those 
participating  in  CATs,  57  have  been  involved  in  only  one,  19 
have  been  involved  in  two,  and  thirteen  have  been  involved 
in  three  or  more. 

A  chi'Square  test  was  performed  to  determine  if  a 
significant  number  of  training  participants  became  involved 
with  corrective  action  teams.  Those  who  have  not 
participated  represented  one  category  and  those  who  were 
involved  represented  a  second  category.  Results  yielded  a 
chi-square  statistic  of  3.56  which  was  small  enough  to 
indicate  an  even  distribution  of  past  participants  that 
both  do  and  do  not  get  involved  in  CATs  after  training. 

Conclusion.  A  significant  number  of  training 
participants  do  not  become  members  of  CATs  after  the 
training  is  completed.  The  number  who  do  get  involved  is 
approximately  the  same  number  as  those  who  do  not  get 
involved. 

Investigative  Question  7 

Is  there  a  noticeable  decline  in  self-efficacy 
regarding  application  of  training  material  as  time 
since  training  increases? 
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This  question  involves  investigating  the  sel£*ef f icacy 
score  trend  of  each  training  group  with  respect  to  the 
length  of  time  between  responding  to  the  questionnaire  and 
attending  CAT  leader  training.  Even  though  samples  were 
independent  making  longitudinal  tests  impossible  with  the 
one-shot  case  study  format,  review  of  self-efficacy  scores 
between  groups  was  accomplished  to  identify  any  possible 
time  related  trends. 

Between  Training  Groups.  A  two  sample  t-test  approach 
was  used  to  determine  if  the  scores  varied  significantly 
from  one  group  of  classes  to  the  next.  To  increase  the  size 
of  each  sample  and  investigate  longer  term  time  trend 
results,  the  twenty-one  classes  were  combined 
chronologically  into  seven  training  groups.  Each  group 
contained  three  classes.  The  class  dates,  group  size,  self- 
efficacy  means  and  standard  errors  for  each  group  are 
identified  in  Table  14. 

A  two-sample  t-test  was  performed  on  every  combination 
of  group  scores  to  determine  if  there  was  a  noticeable 
decline  in  self-efficacy  with  an  increase  in  time  since 
training.  Results  of  all  combinations,  showed  that  only  the 
comparison  between  groups  2  and  5  indicated  a  significant 
difference  in  mean  scores  (p  <  .05).  Visual  inspection  of 
Table  14  confirms  this  result  since  these  two  self-efficacy 
scores  showed  the  widest  variation  between  all  groups. 

Investigating  trends  between  training  groups  provides 
an  indication  that  there  is  not  a  significant  decrease  in 
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Table  14 

Training  Group  Self-efficacy  Scores 


Grout) 

Training 

dates 

N 

Self 

-efficacy  Score 

( final 

day) 

Mean 

Std  Error 

1 

1  Sep 

1989 

30 

24.4 

1.10 

29  Sep 

1989 

13  Oct 

1989 

2 

17  Nov 

1989 

27 

25.8 

0.83 

8  Dec 

1989 

12  Jan 

1990 

3 

9  Peb 

1990 

34 

22.5 

2.05 

9  Mar 

1990 

6  Apr 

1990 

4 

4  May 

1990 

30 

25.1 

1.13 

8  Jun 

1990 

13  Jul 

1990 

5 

20  Jul 

1990 

32 

20.8 

1.66 

10  Aug 

1990 

14  Sep 

1990 

6 

12  Oct 

1990 

18 

23.1 

1.71 

16  Nov 

1990 

7  Dec 

1990 

7 

8  Feb 

1991 

39 

23.7 

0.62 

22  Mar 

1991 

12  Apr 

1991 

self-efficacy  over  a  time  period  of  several  months,  but  it 
does  not  address  the  change  in  self-efficacy  that  occurs 
immediately  after  training.  The  investigation  of  training 
classes  was  briefly  considered  to  examine  this  issue. 
However,  scores  between  classes  had  so  much  variation,  that 
comparisons  between  separate  classes  was  inappropriate.  A 
longitudinal  study  of  the  same  class  over  time  is 
reconvnended  to  investigate  short  term  self-efficacy  changes. 

Conclusion.  Data  does  not  show  that  self-efficacy 
exhibits  a  significant  trend  between  classes  as  the  time 
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since  training  increases.  The  large  range  of  self-efficacy 
scores  and  the  lack  of  longitudinal  data  for  a  specific 
class  do  not  permit  any  conclusions  for  short  term  or  intra¬ 
group  score  impacts  resulting  from  time. 

Investigative  Question  8 

Are  problem  solving  tools  and  techniques  being  used  in 

the  organizational  and  CAT  environments? 

In  Chapter  3,  the  application  of  learning  was 
emphasized  as  the  reason  for  the  training  provided.  Since 
the  problem  solving  steps  and  the  related  tools  and 
techniques  constitute  a  large  portion  of  the  CAT  leader 
training  program,  specific  emphasis  is  being  placed  on 
whether  these  tools  are  being  applied  after  training  has 
occurred.  The  steps  of  problem  solving  were  considered  a 
tool  or  technique  for  this  section  of  the  research. 

Overall  Level  of  Use.  Section  5  of  the  questionnaire 
involved  the  application  of  tools  and  techniques.  Items 
involving  the  levels  of  use  constituted  the  first  portion  of 
this  section.  Respondents  were  requested  to  note  the  level 
of  use  of  the  problem  solving  tools  and  techniques  provided 
in  the  course.  A  summary  version  five-point  Likert  scale 
used  to  measure  responses  in  the  questionnaire  is  identified 
in  Figure  8. 

Table  15  identifies  the  frequency  of  responses,  in  both 
the  CAT  and  organizational  environment,  for  each  of  the  10 
problem  solving  items.  Categories  of  use  correspond  with 
the  Likert  scale  identified  in  Figure  8,  except  that  the  CAT 
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and  organizational  environment  categories  were  combined  to 
limit  the  data  to  three  categories  (not  used,  seen  used,  and 
personally  used). 


Table  15 

Frequency  of  Responses  Regarding  Overall  Level  of  Use 


Tool  or  Techniaue 

Not 

Used 

Seen 

Used 

Personal ly 
Used 

Problem  Solving  Steps 

31 

44 

124 

Brainstorming 

8 

20 

171 

NOT 

54 

47 

95 

Cause  and  Effect 

103 

52 

41 

Pareto  Analysis 

87 

66 

44 

Check  Sheets 

97 

51 

53 

Plow  Charting 

33 

58 

105 

Solution  Sel  Matrix 

116 

48 

31 

Force  Field  Analysis 

127 

41 

34 

Action  Planning 

64 

45 

94 

A  chi-square  test  for  goodness  of  fit  was  performed  on 
all  topics  to  determine  if  the  responses  followed  a 
homogeneous  distribution.  In  every  case,  that  hypothesis 
was  rejected  because  of  a  disproportionate  number  of 
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responses  in  a  specific  category  (p  <  .05).  These  large 
response  categories  represented  two  extremes.  The  first 
involved  five  items  where  a  significant  ntimber  of 
respondents  indicated  they  had  not  seen  the  tool  or 
technique  used  at  all.  The  second  extreme  involved  the 
other  five  items  where  a  significant  number  of  respondents 
indicated  they  had  personally  used  the  tool  or  technique. 
There  was  no  case  where  the  largest  response  concerned 
seeing  a  tool  or  technique  used,  but  not  using  it 
personally. 

Table  16  identifies  those  topics  where  "no  use"  was  the 
most  significant  category  noted  by  the  respondents.  Force 
field  analysis  was  identified  as  the  least  used  of  all  the 
topics  discussed  where  €3%  of  the  respondents  stated  that 
they  have  never  seen  or  used  force  field  analysis  in  either 
a  CAT  or  organisation  setting.  The  other  topics  included: 
cause  and  effect  analysis,  pareto  analysis,  and  the  solution 
selection  matrix. 


Table  16 

Largest  Category  -  Tool  or  Technique  Not  Used 


Tool  or  Techniaue 

Number 

%  of 
total 

Chi“ 

sauare 

Force  Field  Analysis 

127 

63% 

79.7 

Solution  Selection  Matrix 

116 

59% 

62.3 

Cause  and  Effect  Analysis 

103 

53% 

33.5 

Check  Sheets 

97 

48% 

20.2 

Pareto  Analysis 

87 

44% 

14.1 
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Table  17  identifies  those  topics  where  "personal  use" 


was  the  most  significant  category  noted  by  respondents. 
Brainstorming  was  identified  as  the  most  used  of  all  topics 
addressed.  86%  of  the  respondents  stated  that  they  have 
personally  used  brainstorming  in  either  the  CAT  or 
organizational  setting.  The  other  topics  personally  used  by 
a  significant  number  of  those  surveyed  included:  steps  in 
problem  solving,  nominal  group  technique,  flow  diagrams,  and 
action  planning. 


Table  17 

Largest  Category  -  Tool  or  Technique  Used 


%  of 
total 

Chi- 

square 

Brainstorming 

171 

86% 

248.8 

Steps  in  Problem  Solving 

124 

62% 

76.5 

Flow  Diagrams 

105 

54% 

40.9 

Nominal  Group  Technique 

95 

48% 

20.6 

Action  Planning 

94 

46% 

18.0 

Conclusion.  Results  indicate  that  of  the  ten  topics 
that  respondents  addressed,  half  did  not  see  a  significant 
amount  of  use  outside  of  the  classroom,  while  half  of  them 
did.  As  noted  earlier,  the  CAT  leader  training  forum  is  not 
the  only  place  where  individuals  are  exposed  to  the  problem 
solving  tools  and  techniques  discussed.  But  it  is  designed 
to  promote  their  continued  and  effective  use.  From  the  data 
collected,  no  observation  can  be  provided  about  the 
increased  use  of  these  tools  and  techniques  that  result  from 
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training,  since  longitudinal  data  is  not  available. 

However,  the  level  of  current  use  noted  here  can  provide  a 
basis  for  the  focus  of  future  course  offerings. 

Investigative  Question  9 

Are  problem  solving  tools  and  techniques  perceived  as 
being  used  frequently  in  the  CAT  and  organisational 
environment? 

Frequency  of  use  was  another  item  measured  as  part  of 
Section  5  of  the  questionnaire.  Respondents  were  requested 
to  note  the  frequency  of  use  over  the  last  three  years  of 
each  of  the  problem  solving  tools.  The  frequency  of  use 
involved  a  four  point  Likert  scale  that  included;  never, 
once,  twice,  and  three  or  more  times. 

Table  18  identifies  the  frequency  of  responses  for  each 
of  the  ten  problem  solving  items  of  interest. 


Table  18 

Frequency  of  Responses  Regarding  Frequency  of  Use 


Tool  or  Techniaue 

Never 

Once 

Twice 

Three+ 

Problem  Solving  Steps 

41 

32 

28 

101 

Brainstorming 

12 

24 

18 

146 

NOT 

76 

46 

24 

53 

Cause  and  Effect 

128 

40 

22 

13 

Pareto  Analysis 

124 

35 

17 

23 

Check  Sheets 

117 

30 

18 

35 

Flow  Charting 

47 

37 

36 

83 

Solution  Sel  Matrix 

149 

32 

12 

10 

Force  Field  Analysis 

152 

28 

16 

6 

Action  Planning 

77 

33 

17 

73 
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A  chi-square  test  for  goodness  of  fit  was  performed  on 
all  topics  to  determine  if  the  responses  followed  a 
homogeneous  distribution.  In  every  case,  that  hypothesis 
was  rejected  because  of  a  disproportionate  nximber  of 
responses  in  a  specific  category  (p  <  .05). 

Frequency  vs.  Level  of  Use.  Comparison  of  the 
frequency  data  with  the  level  of  use  data  showed  several 
trends.  As  expected,  the  topics  which  were  not  used  by  a 
significant  number  of  people,  also  had  a  corresponding  low 
frequency  of  use.  Additionally,  those  topics  that  were  used 
by  a  significant  number  of  respondents,  also  had  the  highest 
level  of  use  (three  of  more  times). 

Overall  Frequency  of  Use.  All  topics  receive  some  use 
in  the  organizational  environment.  Even  on  those  topics 
which  are  not  used  by  a  significant  number  of  individuals,  a 
minimum  of  50  people  have  indicated  they  have  seen  the  topic 
used  at  least  once  in  the  organization.  As  in  the 
discussion  regarding  level  of  use,  longitudinal  data  is  not 
available  to  draw  conclusions  on  the  impact  of  CAT  leader 
training  on  the  use  of  these  tools  and  techniques  in  the 
field.  However,  this  data  does  show  that  the  tools  and 
techniques  are  used  in  ASD. 

Recent  Use.  Since  these  tools  and  techniques  have  been 
taught  for  years,  one  concern  involved  respondents  who 
indicated  they  had  seen  a  tools  or  technique  used,  but 
during  a  time  period  that  preceded  CAT  leader  training.  To 
determine  the  number  who  had  seen  it  used  recently  and 
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possibly  as  a  result  of  CAT  leader  training  (in  the  last 
three  years),  the  size  of  response  of  those  who  have  never 
seen  a  tool  or  technique  used  outside  training  was  compared 
to  the  size  of  response  of  those  who  had  seen  no  use  in  the 
past  three  years.  The  data  indicated  that  a  portion  of 
those  who  have  seen  a  tool  or  technique  used,  had  not  seen 
it  used  during  the  past  three  years.  This  information  is 
provided  for  each  tool  and  technique  in  Table  19  and  is 
ordered  by  the  largest  percentage  of  respondents  who  have 
not  seen  it  used  recently. 


Table  19 

Respondents  Who  Have  Not  Recently  Seen  a  Topic  Used 


Tool  or  Technioue 

No  use 

No  use 
in  3  yrs 

Chanoe 

%1 

Pareto  Analysis 

87 

124 

37 

19% 

Solution  Sel  Matrix 

116 

149 

33 

17% 

Cause  and  Effect 

103 

128 

25 

13% 

Force  Field  Analysis 

127 

152 

25 

12% 

NOT 

54 

76 

22 

11% 

Check  Sheets 

97 

117 

20 

10% 

Flow  Diagrams 

33 

47 

14 

7% 

Action  Planning 

64 

77 

13 

6% 

Problem  Solving  Steps 

31 

41 

10 

5% 

Brainstorming 

8 

12 

4 

2% 

*  -  %  computed  based 

on  total 

number  of 

respondents 

Pareto  Analysis  ranked  first  out  of  the  topics  in  the 
percentage  of  individuals  who  have  not  seen  a  specific  topic 
used  in  the  last  three  years.  Brainstorming  was  the  lowest 
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topic  where  almost  all  individuals  exposed  to  it,  have  seen 
it  used  in  the  last  three  years.  Since  missing  data  results 
in  different  sample  sizes  for  individual  questions, 
percentages  were  based  on  the  total  number  of  respondents. 

Conclusion .  In  those  cases  where  the  level  of  use  was 
significantly  high,  a  higher  frequency  of  use  was  noted. 
Likewise,  the  least  used  tools  and  techniques  had  the  lowest 
frequency  of  use.  Use  of  all  the  tools  and  techniques  have 
been  witnessed  by  at  least  50  individuals  indicating  that 
these  subjects  are  being  practiced  in  the  field.  For  those 
who  indicated  they  had  not  seen  a  tool  or  technique  used, 
the  large  majority  (over  80%  in  all  cases)  have  seen  it  used 
in  the  last  three  years.  While  this  corresponds  with  the 
with  the  approximate  beginning  of  CAT  leader  training,  the 
specific  impact  of  CAT  leader  training  toward  use  cannot  be 
determined  with  the  case  study  data. 

Investigative  Question  10 

Are  problem  solving  tools  and  techniques  perceived  as 

being  useful  tools  for  team  problem  solving? 

Now  that  the  level  and  frequency  of  use  for  problem 
solving  tools  and  techniques  has  been  discussed,  the 
perception  of  usefulness  will  be  addressed.  Respondents 
were  requested  to  note  the  usefulness  of  the  topics  as  tools 
for  team  problem  solving.  The  responses  involved  a  five 
point  Likert  scale  that  described  items  as:  very  useful, 
somewhat  useful,  undecided,  rarely  useful,  and  not  useful. 
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Table  20  identifies  the  frequency  of  the  level  of  usefulness 
responses  for  each  of  the  items  of  interest. 


Table  20 

Frequency  of  Responses  Regarding  Usefulness 


Tool  or  Techniaue 

1 

2 

3 

4 

5 

Problem  Solving  Steps 

136 

49 

12 

2 

4 

Brainstorming 

158 

34 

7 

1 

4 

NOT 

74 

66 

47 

10 

6 

Cause  and  Effect 

29 

65 

64 

24 

16 

Pareto  Analysis 

39 

69 

61 

19 

10 

Check  Sheets 

40 

65 

70 

15 

9 

Flow  Charting 

82 

72 

30 

8 

11 

Solution  Sel  Matrix 

22 

54 

93 

20 

13 

Force  Field  Analysis 

27 

40 

94 

22 

19 

Action  Planning 

93 

48 

44 

5 

8 

l^Very  useful  2=Somewhat 

4-Rarely  useful 

useful 

5=Not 

3=Undecided 

useful 

A  five-point  Likert  scale  was  used  to  develop  an 
interval  rating  of  the  degree  of  usefulness  of  each  item 
listed.  Since  the  options  represented  a  range  of  responses 
possible,  a  weighted  linear  transformation  was  performed  to 
provide  a  relative  scale  of  usefulness  for  each  item.  The 
scale  went  from  very  useful  (-*'2)  to  not  useful  (~2)  with 
even  intervals  of  one  in  between.  Table  21  provides  those 
results  along  with  the  percentage  of  respondents  who  rated 
the  item  "rarely  useful”  or  "not  useful". 
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Table  21 

Tools  and  Techniques  Transformation  Values  and  Low 
Usefulness  Response  Percentages 


Tool  or  Techniaue 

Ratino 

Rarely  or 

Hot  Useful 

Brainstorming 

1.67 

2.5% 

Problem  Solving 

1.53 

3.0% 

Action  Planning 

1.08 

6.6% 

Plow  Charting 

1.01 

9.4% 

NOT 

0.95 

7.9% 

Check  Sheets 

0.56 

12.1% 

Pareto  Analysis 

0.55 

14.6% 

Cause  and  Effect  Analysis 

0.34 

20.2% 

Solution  Selection  Matrix 

0.26 

16.3% 

Force  Field  Analysis 

0.17 

20.3% 

All  topics  had  a  positive  mean  indicating  that  the 
respondents  found  some  degree  of  usefulness  in  every  topic. 
Brainstorming  and  the  steps  to  problem  solving  were  seen  as 
very  useful  by  the  vast  majority  of  those  surveyed.  Action 
planning,  flow  charting,  nominal  group  technique,  and  pareto 
were  perceived  as  the  next  most  useful  set  of  tools  and 
techniques.  In  this  group,  the  majority  of  respondents 
rated  the  items  as  either  very  or  somewhat  useful .  Check 
sheets,  cause  and  effect  diagrams,  the  solution  selection 
matrix,  and  force  field  diagrams  constituted  the  remaining 
items.  In  three  of  these  four  cases,  the  largest  response 
category  was  "undecided",  indicating  that  many  of  those 
surveyed  had  not  yet  formed  an  opinion  on  the  degree  of 
usefulness  for  those  tools  and  techniques. 
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Rarely  or  Not  Dseful  Ratings.  The  last  colunm  of  Table 
21  provides  the  percentage  of  those  surveyed  who  rated  the 
item  as  "rarely  useful"  or  "not  useful".  This  data  provides 
insight  into  the  perception  of  respondents  regarding  the 
material  provided  in  the  course.  Force  field  analysis  and 
cause  and  effect  diagrams  represented  items  that  had  the 
largest  number  of  low  ratings.  However,  in  all  cases,  the 
majority  of  respondents  rated  the  tool  or  technique  useful 
or  was  at  least  undecided.  This  indicates  that  no  tools  or 
technique  taught  in  the  course  was  perceived  as 
inappropriate  for  use  in  the  organisational  environment. 

Conclusion.  The  transformation  ratings  indicate  that 
all  tools  and  techniques  have  some  degree  of  usefulness  as 
problem  solving  tools.  While  some  items  (Check  sheets, 
pareto  analysis,  cause  and  effect  diagrams,  the  solution 
selection  matrix,  and  force  field  analysis),  have  a  large 
number  of  "undecided"  responses,  no  item  was  found  to  have 
no  use  in  problem  solving. 

Investigative  Question  11 

Do  a  significant  number  of  CAT  leaders  rate  the  use  of 
a  CAT  as  appropriate  in  addressing  the  CAT  related 
issues  assigned? 

CATs  are  often  assigned  based  on  the  "type"  of  issue 
involved.  Chapter  2  described  three  categories  that  were 
used  to  aid  in  assigning  and  tracking  issues  to  the 
appropriate  owners  and  organizations.  Type  2  issues 
involved  coordination  by  several  offices  in  an  organization 
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and  normally  is  a  prime  candidate  for  a  corrective  action 
team  (ASD/TQ,  undated:10).  This  investigative  question 
involves  whether  the  respondents  involved  with  a  CAT 
perceive  its  use  a  appropriate  in  addressing  the  issue. 

Forty-six  of  the  respondents  who  were  participating  in 
a  CAT  rated  it  as  very  applicable  for  the  issue  involved,  33 
rated  it  somewhat  applicable,  9  rated  it  as  not  very 
applicable,  and  6  provided  a  rating  of  not  applicable. 

These  categories  were  combined  into  two  groups  with  very  and 
somewhat  applicable  making  up  the  first  group  and  not  very 
and  not  applicable  making  up  the  second. 

A  chi-square  test  was  performed  to  determine  if  the 
frequency  of  responses  for  these  two  groups  followed  a 
homogeneous  distribution,  indicating  at  least  half  of  the 
respondents  did  not  see  the  use  of  a  CAT  as  appropriate. 
Results  yielded  a  chi-square  statistic  of  68.8  which  was 
sufficiently  large  to  reject  the  homogeneous  distribution 
test . 

Conclusion.  A  significant  number  a  past  training 
participants  who  have  been  involved  in  at  least  one  CAT 
perceive  the  use  of  a  CAT  as  appropriate  in  addressing  the 
issue  assigned. 

Questionnaire  Written  Comments 

Section  6  of  the  questionnaire  solicited  feedback 
regarding  areas  not  covered  or  insufficiently  covered  by 
previous  items.  Respondents  were  asked  to  use  their 
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experience  as  graduates  of  CAT  leader  training  who  are  now 
involved  in  organizational  issues  to  suggest  course  changes 
and  improvements. 

Nine  categories  for  comments  were  provided  to  encourage 
specific  feedback  on  each  facet  of  the  training.  These 
categories  included:  TQ  background,  personal  values,  group 
dynamics,  steps  of  problem  solving,  tools  and  techniques, 
exercises,  movies,  final  exercise,  and  other.  Sixty-nine  of 
those  surveyed  provided  written  comments.  Appendix  G 
contains  those  written  comments  broken  out  by  topic.  The 
overall  comments  provided  by  respondents  will  first  be 
discussed  followed  by  feedback  on  specific  sections. 

Only  those  comments  that  have  universal  application  to 
the  current  syllabus  will  be  discussed  in  this  chapter.  The 
summaries  below  are  designed  to  overview  the  responses  made 
by  course  graduates.  Appendix  G  represents  the  source  of 
those  summaries. 

Overall  comments.  Fifty-two  comments  that  were 
classified  as  overall  (or  other)  were  made  by  respondents. 
Twenty-four  positive  overall  comments  were  made  in  this 
section  compared  with  four  negative.  Most  positive  comments 
addressed  the  respondent's  agreement  on  the  current  course 
content  and  delivery.  Negative  comments  involved  the  lack 
utilization  as  a  CAT  leader  training  graduate,  length  of  the 
training  (too  long),  the  lack  of  follow-up,  and  the  need  to 
discuss  more  ASD  related  issues. 
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The  main  recommendations  involved  increasing  the 
trained  population  to  include  all  personnel,  developing  a 
refresher  course  and/or  making  training  more  timely, 
providing  periodic  surveys  to  past  attendants,  and  emphasis 
on  team  dynamic  issues  involving  conflict  and  management 
resistance . 

Total  Quality  Comments.  Eighteen  comments  represented 
the  range  of  responses  on  the  TQ  background  section.  The 
majority  of  those  making  reconunendations  indicated  that  the 
material  needs  to  be  tailored  to  the  individual  group.  Many 
recognized  the  need  for  information  on  total  quality,  but 
stated  that  it  was  redundant  for  some  of  the  participants. 
Those  with  minimal  TQ  training  stated  that  more  of  the 
basics  were  needed. 

Negative  comments  involved  too  much  time  spent  on 
fundamentals.  Additionally,  some  of  the  material  was 
described  as  outdated  and  too  narrow  in  focus  since  it 
didn't  include  more  discussion  of  individuals  such  as 
Deming,  Crosby,  etc. 

Personal  Values  Comments.  Of  the  twelve  comments  on 
the  personal  values  section,  ten  were  overwhelmingly 
positive.  Several  of  those  who  responded  indicated  it  was 
helpful  both  in  understanding  themselves,  and  those  arotind 
them.  A  recommendation  by  one  past  attendee  included  using 
the  Myers-Briggs  Type  Indicator  as  a  substitute  for  the 
Strength  Deployment  Inventory. 
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Group  Dynamic  Comments.  Seven  individuals  responding 


with  conunents  on  group  dynamics  were  mostly  favorable. 
Several  identified  the  need  to  spend  more  time  on  the 
subject  and  include  as  a  stronger  focus  on  what  makes  a 
successful  team  and  a  recommendation  to  use  role  playing  in 
learning  to  deal  with  difficult  team  members. 

These  comments  are  supported  by  the  negative  content 
validity  ratio  (CVR)  related  to  dealing  with  difficult  team 
members  that  was  discussed  in  Chapter  4. 

Steps  of  Problem  Solving  Comments.  The  smallest  group 
size  of  six  provided  comments  for  problem  solving  steps. 
Overall,  the  comments  were  positive  and  two  of  the  six  who 
responded  recommended  a  summary  handout  of  the  related 
material  for  use  as  a  reference. 

Tools  and  Techniques  Comments.  Sixteen  separate 
comments  were  provided  for  the  tools  and  techniques. 
Recommendations  included  focusing  on  the  most  used  items, 
updating  the  manual ,  and  providing  more  information  on  the 
when  to  use  specific  tools  and  techniques.  Again,  a  summary 
handout  was  recommended  of  the  material  for  use  as  a 
reference . 

Exercise  and  Movie  Comments.  Nine  individuals 
responded  with  comments  and  suggestions  on  the  use  of 
exercises  in  the  course.  The  most  frequent  response 
involved  the  need  to  make  the  exercises  more  relevant  to  the 
work  environment. 
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Eight  responses  were  provided  to  the  movies  category. 
One  respondent  thought  they  were  not  useful  while  another 
indicted  they  were  redundant  because  he/she  had  seen  some  in 
other  training  classes.  The  "Paradigm”  film.  Meeting 
Robbers,  and  The  Abilene  Paradox  were  all  specifically 
referenced  as  great  movies. 

Final  Exercise  Comments.  Fifteen  comments  were 
collected  on  the  final  exercise.  Comments  mostly  involved 
the  need  for  a  more  relevant  subject.  Those  with  opinions 
on  use  of  a  final  exercise  to  reinforce  material  learned 
during  the  week  were  positive  on  that  approach.  Comments 
were  also  made  on  the  time  wasted  while  waiting  to  collect 
information  from  facilitators  and  the  problems  associated 
with  being  located  too  closely  to  other  groups. 

The  written  comments  provided  by  those  surveyed  were 
generally  positive  and  involved  many  recommendations  that 
may  be  used  to  improve  the  course.  Several  individuals 
provided  typed  comments  to  this  section  indicating  the 
investment  of  both  time  and  thought  in  responses. 

Summary 

Chapter  5  provided  a  discussion  of  the  data  collected 
from  the  objectives,  the  knowledge  test,  and  the 
questionnaire  and  formally  concludes  the  requirements  of  the 
first  five  of  the  six  major  research  objectives. 
Investigative  questions  identified  in  Chapter  1  were 
addressed  and  conclusions  were  drawn  for  each.  In  addition. 
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written  questionnaire  comments  were  discussed  and  summarized 
at  the  end  of  the  chapter. 

Chapter  6  now  addresses  the  final  research  objective 
involving  recommendations  for  improvements  to  the  CAT  leader 
training  curriculum  based  on  the  information  just  discussed. 
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VI.  Conclusions  and  Recommendations 


Introduction 

The  developed  objectives,  knowledge  test  results  and 
questionnaire  responses  provide  a  basis  for  a  comprehensive 
study  into  the  effectiveness  of  CAT  leader  training.  The 
conclusions  and  recommendations  that  follow,  are  based  on  a 
combination  of  the  results  in  chapter  5  and  the  experience 
of  the  researcher  in  ASD  total  quality  initiatives  and  CAT 
leader  training. 

A  discussion  regarding  how  the  six  research  objectives 
were  addressed  is  given  below.  General  conclusions  are  then 
provided  to  summarize  the  study  results.  Recommendations 
relating  to  changes  in  the  current  course  and  follow-on 
research  ideas  are  then  discussed. 

Meeting  Research  Objective  Requirements 

Chapter  1  identified  the  six  research  objectives  for 
the  study.  These  were  followed  by  investigative  questions 
that  focused  the  effort  and  allowed  individual  hypothesis  to 
be  tested  and  discussed.  The  study  addressed  each  research 
objective.  Validating  a  set  of  proposed  objectives 
(objective  1)  provided  a  framework  for  a  knowledge  test  that 
measured  learning  and  self-efficacy  in  CAT  leader  training 
(objective  2).  Identifying  the  characteristics  of  current 
CATs  and  the  involvement  of  past  course  attendees  was 
measured  by  a  questionnaire  (objective  3).  The  results  of 
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this  questionnaire  also  addressed  both  the  self-efficacy  of 
past  graduates  and  their  perception  and  application  of 
problem  solving  tools  and  techniques  (objectives  4  and  5). 
This  chapter  now  draws  conclusions  involving  the  study,  and 
then  addresses  the  final  research  objective  involving 
recommendations  to  the  Commander  of  Aeronautical  Systems 
Division  (ASD)  for  improvement  of  the  current  CAT  leader 
course  (objective  6). 

General  Conclusions 

General  results  from  the  evaluation  indicate  that  the 
training  is  accomplishing  the  majority  of  its  desired 
behavioral  objectives  and  that  training  related  material  is 
being  used  in  by  graduates.  Below  are  general  conclusions 
on  several  aspects  of  the  evaluation. 

The  Internal  Evaluation.  Results  from  the  internal 
evaluation  indicate  that  learning  has  occurred  with  the 
majority  of  training  material  presented  in  the  course. 
Self-efficacy  increased  in  most  cases  showing  an  improved 
sel f -perception  to  perform  training  related  tasks. 

Predictors  for  a  pre-test  score  model  were  identified,  but 
provided  limited  explanatory  power  to  predict  a  student's 
pre-score. 

Limited  focus  for  the  research  was  placed  on  the  areas 
of  personal  values  and  team  dynamics  mainly  because  of  the 
need  to  limit  the  scope  of  study.  However,  many  of  the 
written  responses  placed  a  strong  emphasis  on  these  subjects 


in  the  course.  This  points  to  their  strong  influence  on  the 
effectiveness  of  the  training  and  the  need  for  future  (and 
more  thorough)  evaluation  in  this  area. 

The  Field  Evaluation.  The  field  evaluation  showed  that 
most  of  the  CAT  leaders  are  assigned  an  issue  within  three 
months  of  training  and  consider  the  issues  worked  relevant 
to  a  group  problem  solving  approach.  Half  of  the  ten  tools 
and  techniques  taught  in  training  were  used  by  the  majority 
of  graduates,  while  the  other  half  saw  little  use.  This 
result  corresponds  with  the  frequency  of  use  for  tools  and 
techniques.  All  the  tools  and  techniques  in  the  current 
curriculum  are  perceived  as  having  at  least  some  degree  of 
usefulness  by  those  who  responded. 

Only  56  of  208  respondents  (27%)  have  become  CAT 
leaders  in  their  organizations.  This  should  be  a  major 
concern  if  the  primary  goal  is  to  develop  trained  personnel 
for  this  purpose.  Current  terminal  objectives  provided  in 
the  CAT  leader  notebook  support  this  purpose;  however, 
recent  discussions  have  disclosed  a  plan  to  change  the 
training  title  to  "Team  Leader  Training"  which  would  place 
less  of  an  emphasis  on  TQ  affiliation  (Wissman,  1991). 

The  Use  of  Specific  Tools  and  Techniques.  The  high 
level  and  frequency  of  use  for  several  tools,  especially 
brainstorming  and  flow  diagrams,  should  be  seen  as  a 
positive  result  shown  by  this  research.  However,  this 
situation  may  exist  because  many  students  have  been  trained 
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in  these  techniques  prior  to  CAT  leader  training. 
Consequently,  this  training  should  eliminate  or  reduce  the 
emphasis  on  these  techniques.  The  growth  in  use  for  all 
tools  and  techniques  should  be  encouraged,  but  not  mandated. 
While  some  tools  and  techniques  have  seen  limited  use  by 
graduates  of  CAT  leader  training,  it  does  not  necessarily 
mean  that  they  deserve  more  use.  CAT  leader  training  should 
ensuring  that  they  are  understood  and  used  properly. 

Many  tools  and  techniques  are  provided  in  the  course  so 
that  options  are  available  to  appropriately  address  a  myriad 
of  organizational  issues.  However,  the  study  shows  that 
many  students  have  not  demonstrated  an  ability  to  identify 
the  type  of  data  involved  (quantitative  or  qualitative)  and 
the  appropriate  tool  or  technique  for  a  specific  problem 
solving  step.  This  knowledge  is  essential  for  effective 
application  of  the  material  in  a  CAT. 

Terminal  Objectives.  The  terminal  objectives  are  the 
three  major  objectives  currently  identified  in  the  CAT 
leader  notebook.  To  summarize,  they  include  ensuring 
graduates  are  able  to:  1)  organize  and  manage  a  CAT  to 
become  and  remain  an  effective  functioning  team,  2)  explain 
the  TQ  structure  and  process  effectively,  and  3}  teach  the 
problem  solving  methodology  and  the  appropriate  tools  and 
techniques  to  CAT  members. 

The  study  indicates  that  students  are  learning  the 
material  and  important  methods  to  develop  and  run  an 
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effective  CAT.  However,  timeliness  of  training  and  the 
small  amount  of  feedback  from  graduates,  currently  limits 
the  ability  of  training  to  be  even  more  effective. 

Regarding  objective  1  that  involves  organizing  and 
running  and  effective  CAT;  only  56  of  208  (27%)  respondents 
to  the  questionnaire  indicated  they  have  performed  as  a  CAT 
leader.  This  data  shows  that  the  majority  of  graduates  do 
not  fill  this  role.  If  developing  CAT  leaders  is  a  primary 
objective,  then  efforts  to  promote  more  graduates  into  these 
positions  is  required.  This  may  include  educating 
organizations  in  their  use,  modifying  the  selection  criteria 
for  training  and  perhaps  introducing  changes  in  the  course 
format  to  include  even  more  specific  CAT  leader  exercises 
and  examples. 

For  the  second  objective  involving  explaining  the  TQ 
process,  the  roles  and  membership  of  the  TQ  structure  is 
understood  by  most  individuals.  Students  have  learned  the 
material;  however,  their  ability  to  synthesize  and  explain 
the  process  represents  a  higher  level  of  comprehension  than 
that  measured  in  this  study. 

Finally,  the  ability  of  graduates  to  teach  group 
problem  solving  and  team  dynamics  material  is  still  an  open 
issue.  This  study  showed  that  the  material  was  understood 
via  knowledge  test  results,  but  a  far  more  complicated 
measurement  is  required  to  demonstrate  the  ability  to  teach 
the  material.  Indeed,  the  conclusion  that  previous  CAT 
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experience  is  not  significantly  related  to  a  pre-test  score 
supports  the  possibility  that  CAT  leaders  do  not  train 
members  to  use  the  problem  solving  tools.  Review  of  the 
entire  course  seems  to  show  a  lack  of  congruency  regarding 
this  objective.  Class  discussions  and  exercise  do  not 
openly  support  this  final  objective.  The  researcher's 
observation  is  that  students  are  taught  to  use  the  tools  and 
techniques--they  are  not  taught  to  teach  them. 

Recommended  Chances 

Recommended  changes  are  supplied  while  acknowledging 
the  limitations  and  advantages  of  an  outside  perspective. 
Clearly,  none  of  the  ideas  provided  should  be  made  without 
the  understanding  and  concurrence  of  the  course 
facilitators.  These  individuals  have  a  wealth  of  knowledge 
and  experience  that  cannot  be  documented  within  these  pages. 
They  are  responsible  for  the  successes  of  the  training  to 
date.  Recommendations  for  their  consideration  are  provided 
bel ow. 

Review  Course  Terminal  Objectives.  As  stated  in  the 
general  conclusions  section,  both  the  first  and  third 
terminal  objectives  are  not  completely  met  by  the  current 
course.  The  first  objective  of  training  CAT  leaders  is  not 
met  since  only  27%  of  graduates  go  on  to  fill  that  role. 

The  third  objective  involving  the  student's  ability  to  teach 
the  material,  is  not  supported  by  the  current  training 
methods.  Either  the  training  methods  need  to  be  changed,  or 
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the  terminal  objectives  need  to  be  modified.  The  second 
terminal  objective  coincides  with  the  current  training 
program,  but  should  still  be  reviewed  to  determine  if  it 
remains  appropriate.  All  objectives  should  be  considered 
with  respect  of  the  organizational  needs  currently  present. 

Incorporate  Training  Block  Objectives.  Training 
behavioral  objectives  were  developed  as  part  of  this 
research  and  should  be  incorporated  into  the  syllabus. 

These  objectives  formally  describe  both  the  material  and 
level  of  learning  desired  by  the  current  syllabus.  They 
communicate  expectations  to  the  learner  and  provide  a 
standard  of  measurement  for  training  effectiveness 
(Goldstein,  1984:59). 

Changes  in  course  structure  and  content  should  consider 
current  emphasis,  overall  course  constraints,  and  terminal 
objectives.  Clearly,  if  adding  additional  material  is 
considered,  then  a  corresponding  decrease  in  the  current 
syllabus  or  an  increase  in  course  length  is  required.  The 
behavioral  objectives  provide  a  guideline  to  accomplish  this 
task . 

Investigate  Areas  Showing  Little  Improvement.  When 
reviewing  areas  to  focus  improvement  from  knowledge  test 
data,  facilitators  should  investigate  any  area  where  a 
significant  improvement  in  score  was  not  made  by  all 
classes.  While  unique  characteristics  exist  among  each 
class,  different  conditions  introduced  by  facilitators  may 
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have  an  impact  on  performance  and  effectiveness.  If  this  is 
the  case,  then  they  need  to  be  identified  and  addressed. 

Review  Objectives  Not  Emphasized.  Seven  of  the 
proposed  objectives  had  negative  content  validity  ratio 
(CVR)  scores.  This  indicates  that  there  is  material  in  the 
CAT  leader  notebook  that  is  not  being  taught  by  the  majority 
of  instructors  during  the  class.  This  would  be  expected 
since  the  CAT  leader  notebook  holds  substantially  more 
information  than  can  be  provided  in  a  four-day  course. 

All  objectives  have  a  CVR  score  that  provides  a 
relative  scale  of  emphasis  by  instructors.  It  would  be 
useful  in  reviewing  and  in  determining  the  desired  mix  in 
course  content.  A  formal  decision  on  any  change  in 
priorities  should  be  discussed  and  agreed  upon  by  both 
facilitators  and  the  course  director.  Field  units  and 
current  CAT  leaders  should  also  be  queried  to  identify 
current  system  requirements. 

In  addition  to  the  use  of  content  validity  ratios,  the 
knowledge  test  data  regarding  comprehension  of  specific 
tools  and  techniques  may  be  useful  to  identify  those 
subjects  understood  by  students  prior  to  the  class  (pre-test 
results).  This  information  may  allow  the  facilitators  to 
de-emphasize  some  material  by  providing  a  summary  of  the 
relative  information.  This  would  allow  more  time  for  other 
areas.  The  use  of  the  instructional  systems  development 
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(ISD)  model  would  be  an  excellent  tool  to  coordinate  this 
process . 

Measure  Student  Sel f-Ef f icacv.  The  five  question  scale 
developed  in  this  research  covers  the  main  areas  of  task 
transference  in  the  course  related  to  personal  use  and 
application.  This  overall  score  can  be  computed  for  each 
class  to  provide  an  indication  of  the  level  of  confidence  it 
has  in  accomplishing  course  related  tasks. 

Easily  computed  averages  for  each  class  the  first  day 
may  provide  some  indications  as  to  the  level  of  self- 
confidence  of  each  class,  allowing  facilitators  the  ability 
to  tailor  their  presentations.  Additionally,  a  more 
quantitative  measurement  (as  opposed  to  a  critique)  of 
training  effectiveness  could  be  provided  from  a  pre¬ 
test/post-test  comparison.  Self-efficacy  measurements 
dealing  with  higher  levels  of  transference  (i.e.  teaching  or 
explaining)  could  also  be  useful  in  future  evaluations. 

Results  of  self-efficacy  measurements  in  the  research 
data  indicated  that  the  June  class  showed  the  least  (if  any) 
improvement.  Their  corresponding  scores  in  the  knowledge 
test  also  showed  a  lower  levels  of  learning,  especially 
related  to  tools  and  techniques.  These  results  may  be  the 
result  of  ineffective  instruction,  improper  test  measurement 
constructs,  or  environmental  conditions.  Self-efficacy 
scores  related  to  future  test  results  may  provide  more 
insight  into  this  problem. 
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Develop  a  Tools  and  Technique  Workbook.  A  tools  and 
techniques  workbook  should  be  developed.  This  workbook 
could  provide  more  work-related  examples,  an  in-depth 
discussion  of  data  types  and  collection,  and  summary 
information  for  reference  during  meetings. 

Many  of  the  comments  collected  by  past  participants 
involve  the  use  of  problem  solving  tools  and  techniques. 
Specific  comments  involved  knowing  when  to  use  a  specific 
item,  a  desire  for  more  work-related  examples,  and  the  need 
for  a  summary  handout  to  reference  in  meetings.  These 
comments  and  the  researcher's  review  of  the  current  CAT 
leader  notebook  support  the  need  for  a  more  comprehensive 
set  of  references  regarding  specific  uses  of  tools  and 
techniques.  Additionally,  current  CAT  leaders  in  ASD 
organisations  would  be  prime  candidates  for  identifying  and 
developing  organizationally  related  examples. 

One  final  point  that  supports  a  tools  and  techniques 
workbook  involves  the  application  of  material.  Knowledge 
test  results  indicated  a  lack  of  understanding  between  the 
concepts  of  qualitative  versus  quantitative  data.  This 
understanding  is  critical  to  the  proper  use  of  tools  and 
techniques  and  needs  to  be  emphasized  more  both  in  class  and 
in  reference  material . 

Investigate  Training  Format  and  Improve  Follow-up. 
Comments  made  by  several  of  the  past  participants  involved 
the  excessive  length  of  training  and  the  need  for  a 
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refresher  (or  follow-up)  course  for  participants. 

Currently,  time  gaps  in  training  often  occur  for  each  class 
in  different  areas  of  the  course.  Videos  and  exercises  are 
used  well  by  facilitators  to  compensate  for  the  different 
needs  and  schedules  of  each  group.  Unfortunately,  a  four- 
day  course  requires  a  significant  commitment  from  ASD/TQ 
personnel  which  conflicts  with  the  desire  to  offer  a 
refresher  (or  follow-up)  course. 

If  an  objective  of  CAT  leader  training  is  to  continue 
producing  the  current  number  of  graduates,  then  a  one  week 
CAT  seminar  is  recommended  for  study  to  address  training 
length  and  follow-up.  The  one-week  program  would  include 
two  sessions:  the  first  involving  initial  training,  and  the 
second  as  a  refresher  course.  The  initial  training  course 
would  last  three  days  and  constitute  the  majority  (but  not 
all)  of  the  current  course  emphasis.  The  second  session 
would  involve  specific  work  issues  from  ASD.  The  second 
session  could  include  several  categories  of  participants 
that  include:  members  of  the  first  session  that  have 
already  been  assigned  a  CAT  issue,  past  participants 
recently  assigned  an  issue,  leaders  experiencing  problems 
with  their  current  CAT,  personnel  with  past  experience  and 
training  in  the  area,  and  others  desiring  more  exposure  to 
tools  and  techniques. 

Small  groups  can  provide  support  in  determining 
approaches  for  addressing  specific  problems  and  tools  and 
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techniques  to  use.  Complete  CATs  may  be  able  to  use  this 
time  as  an  off-site  to  begin  their  own  group  with  necessary 
support  from  roaming  facilitators  during  those  two  days. 

Such  a  program  would  require  a  great  deal  of  flexibility 
from  the  facilitators,  but  would  address  many  concerns  felt 
by  graduates  who  have  no  formal  follow-on  program. 
Additionally,  it  would  require  only  one  additional  day  in  a 
training  location  (although  more  time  planning). 

Follow-on  Research 

As  discussed  earlier,  this  research  involved  both  an 
internal  and  a  field  evaluation  of  CAT  leader  training; 
however,  the  limited  time  frame  and  lack  of  secondary  data 
limited  the  scope  of  the  effort.  Specifically  in  the  field 
evaluation,  the  one  shot  case-study  approach  precluded  any 
conclusions  about  training  impacts  to  the  organization. 

Below  are  several  ideas  for  follow-on  research  to  build  on 
the  body  of  knowledge  already  developed  and  account  for 
limitations  in  the  research. 

Study  the  Current  Training  Syllabus.  This  research  did 
not  specifically  address  the  appropriateness  of  the  current 
training  syllabus.  Course  material  was  assumed  to  be 
necessary  for  training  a  CAT  leader. 

One  recommended  follow-on  effort  involves  the  use  of 
the  Air  Force  instructional  systems  development  (ISD)  model 
in  reviewing  the  course  from  a  system  requirements 
perspective.  Steps  1  and  2  of  the  ISD  model  involves 
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analyzing  system  (organizational)  requirements  and  then 
defining  a  training  requirement.  This  has  been  accomplished 
for  the  current  program  in  a  less  formal  way  and  would  be  a 
good  basis  for  validating  the  current  syllabus  and 
identifying  necessary  changes.  The  organizations  involved 
in  CAT  leader  training  have  many  unique  requirements  that 
may  require  changes  in  the  focus  of  CAT  leader  training  and 
the  introduction  of  additional  material.  In  the  April 
class,  the  concept  of  storyboarding  was  introduced.  Perhaps 
this  tool  and  others  could  be  reviewed  for  applicability  to 
CAT  leader  training  and  appropriately  incorporated.  The  ISD 
model  provides  and  excellent  framework  to  accomplish  this 
purpose . 

Develop  a  Pre-test  Predictor  Model.  Currently, 
training  is  provided  to  a  broad  range  of  personnel  with 
different  levels  of  experience  and  competence.  A  goal  would 
be  to  tailor  each  course  (or  provide  two  or  more  levels)  to 
meet  the  needs  of  individual  classes  and  allow  facilitators 
to  proceed  at  a  comfortable  pace  for  all  members. 

The  regression  analysis  performed  indicated  that  there 
was  a  noticeable  difference  in  pre-test  scores  based  on  the 
level  of  education  and  past  training  experience  of  an 
individual.  However,  these  predictors  are  not  sufficient  to 
accurately  predict  an  overall  score.  If  future  course 
offerings  are  planned  to  tailor  course  material,  then  a  more 
comprehensive  predictor  model  is  required.  Predictors  that 


include  more  specific  demographic,  training,  and  time 
relative  information  are  required  in  the  model  and  could  be 
useful  in  explaining  a  larger  portion  of  the  score  variance. 

Perform  a  Longitudinal  Study  of  Course  Performance. 

The  pre-test/post-test  data  collection  was  accomplished 
within  three  days.  Even  though  the  control  group  indicated 
no  significant  impact  that  resulted  from  test  sensitization, 
the  short  time  span  does  not  address  the  retention  of 
material  in  the  time  period  following  the  training. 
Administering  the  same  knowledge  test  to  recent  graduates  of 
CAT  leader  training  should  provide  some  insight  into  this 
area . 

If  the  knowledge  test  approach  is  attempted  in  the 
future,  recommend  that  several  questions  are  developed  for 
each  variable  of  interest.  Some  limitations  to  this  study 
were  the  result  of  only  two  or  three  test  questions  per 
variable.  This  limited  the  discrete  distribution  categories 
making  it  more  difficult  to  approximate  the  normal 
distribution . 

Perform  a  Longitudinal  Study  on  Tools  and  Techniques. 
The  use  of  a  one-shot  case  study  as  the  questionnaire 
research  design  does  not  provide  insight  into  the  culture 
change  involving  the  use  of  specific  tools  and  techniques. 
However,  it  does  provide  a  basis  for  future  research  in  this 
area.  Additional  data  collection  from  the  respondents  may 
show  time  related  trends  indicating  the  use  and  perceived 
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usefulness  of  individual  tools  and  techniques.  Since  the 
data  from  the  questionnaire  allows  grouping  by  training 
class,  it  would  be  very  useful.  Also,  data  collection  from 
the  ASD  population  as  a  whole,  can  provide  information 
regarding  the  perceptions  and  actions  of  past  course 
graduates  with  respect  to  those  not  exposed  to  the  training. 

Evaluate  Other  Courses.  This  research  represents  the 
most  comprehensive  evaluation  of  a  quality  related  course  in 
ASD.  Several  other  courses  provided  by  ASD/TQ  have  not 
received  either  an  internal  or  field  evaluation.  Although, 
these  courses  have  not  been  reviewed  as  part  of  thi^ 
re  earch,  many  of  the  work  accomplished  here  may  be 
beneficial  to  other  existing  classes.  Specific  research 
emphasis  recommended  for  other  courses  includes  developing 
behavioral  objectives,  measuring  self-efficacy  of  course 
related  tasks,  identifying  the  level  of  learning  that 
occurs,  and  investigating  field  use  and  acceptance  of  course 
materials . 
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Appendix  A:  CAT  Leader  Training  Syllabus 


Day  1 

I.  INTRODUCTION . 8:00-12:00 

PURPOSE  -  EXPECTATIONS 
LEARNING  OBJECTIVES 
NORMS 

ADMINISTRIVIA 

II.  OVERVIEW  OF  TQ 

HISTORY  OF  TQ  IN  ASD 
STRUCTURE  AND  PROCESS  OF  TQ 

III.  PERSONAL  VALUES 

PERSONAL  VALUES 
MOTIVATION 
PERSONAL  STRENGTHS 


LUNCH . 12:00-1:00 

IV.  GROUP  DYNAMICS . 1:00-4:00 

STAGES  IN  TEAK  FORMATION 
LEADER  STYLE 

CONSENSUS  DECISION  MAKING 

Day  2 


REVIEW  OF  DAY  1 

V.  STEPS  IN  GROUP  PROBLEM  SOLVING.  .  .  .  8:00  -  12:00 
OVERVIEW  OF  STEPS 
OVERVIEW  OF  TECHNIQUES 
MATRIX 

VI.  SELECT  AND  DEFINE  THE  PROBLEM 
BRAINSTORMING 
NOMINAL  GROUP  TECHNIQUE 

VII.  GATHER  INFORMATION 

CAUSE  AND  EFFECT  DIAGRAM 
CHECKSHEETS 


LUNCH . 12:00-1:00 

VIII.  GROUP  DYNAMICS . 1:00-4:00 


DATA  DISPLAY  TECHNIQUES 
PARETO  ANALYSIS 
FLOW  CHARTING 
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Day  3 


REVIEW  OF  DAY  2 

IX.  EVALUATE  ALTERNATE  SOLUTION . 8:00  -  12:00 

SOLUTION  SELECTION  MATRIX 

X.  PLAN  THE  IMPLEMENTATION 

FORCE  FIELD  ANALYSIS 
ACTION  PLANNING 

XI.  IMPLEMENT  THE  SOLUTION 

LONG  TERM  MEASUREMENT 


LUNCH . 12:00-1:00 

XII.  MEETING  EFFECTIVENESS  .  1:00-4:00 

MEETING  ROBBERS 


MANAGEMENT  PRESENTATIONS 

XIII.  ISSUES  AND  ANSWERS 

XIV.  PREPARATION  FOR  APPLICATION  DAY 
Day  4 

XV.  TEST  OF  FIRE  (FINAL  EXERCISE) 


125 


Appendix  B:  CRT  Leader  Training  Proposed  Objectives 


DEPARTMENT  OF  THE  AIR  FORCE 

HCAOOUAnTCMC  ACAONAUTICAI.  aVSTCMS  IMVISION  (AFSC) 
WIIIOHT.AATTEmOM  AIM  FOHCC  MASC.  OHIO  4MJ1-AM3 


AFIT/LSG  (Captain  Streitnater) 


25  Mar  91 


Evaluation  of  Corrective  Action  Team  (CAT)  Leader  Training 


CAT  Leader  Training  Facilitators 

I  am  currently  evaluating  the  CAT  leader  training  provided  by  ASD.  Part  of  this 
evaluation  involves  the  development  of  a  knowledge  test  for  the  course  and  a 
survey  of  past  attendees.  The  knowledge  test  is  designed  to  identify  a 
participant's  aiastery  of  course  objectives  through  examining  what  they  knew 
before  the  course  and  what  they  learned  as  a  result  of  the  course.  The  survey 
is  designed  to  identify  how  course  material  is  used  in  the  organizational 
environmnt.  Both  tools  will  be  useful  in  making  future  course  improvements. 

A  set  of  validated  objectives  is  required  to  ensure  the  development  of  a 
reliable  knowledge  test  and  survey.  Because  of  the  different  training 
approaches  of  each  facilitator,  your  review  is  required  to  ensure  these 
objectives  are  applicable  to  the  course.  Some  objectives  mav  only  relate  to 
material  in  the  CAT  leader  notebook,  but  your  participation  is  necessary  to  see 
what  is  addressed  in  the  discussions.  You  do  not  need  to  identify  yourself 
since  only  group  statistics  are  required. 

Please  read  the  instructions  and  return  the  completed  package  to  the  TQ  office 
NLT  28  Mar  1991.  My  goal  is  to  have  a  knowledge  test  prepared  for  the  April  CAT 
training  course.  If  you  have  any  comients  or  concerns  regarding  the  attached 
objectives,  please  identify  them  in  the  space  following  the  objectives.  Thank 


5  Atch 

1.  Instructions 

2.  List  of  Objectives 

3.  APIT  Ponn  IIC 

4.  List  of  Definitions 

5.  Return  envelope 


» 


126 


Attachment  1:  Instructions 

1.  Review  each  of  the  attached  objectives  (Attachment  2)  and  provide  the 
response  on  the  APIT  Form  IIC  that  is  most  applicable  to  the  way  you  normally 
facilitate  the  course.  Use  the  scope  definitions  below  to  gauge  the  knowledge 
level  required  for  each  objective. 

Scope  Definitions 

The  following  statements  identify  the  scope  of  required  knowledge 
proposed  to  each  objective  and  relate  specifically  to  developing 
questions  for  the  knowledge  test. 

Have  an  appreeiation  fort 

Be  exposed  to  a  concept,  but  not  required  to  recount  it. 

Identify: 

Have  the  ability  to  select  the  answer  from  a  group  of  options,  or 
properly  order  a  list  of  items. 


Know: 


Have  the  ability  to  properly  describe  a  process  or  provide  a 
definition . 

2.  After  responding  to  each  objective  using  the  APIT  Form  IIC,  follow 
directions  for  Sections  2,  3,  and  4  on  the  last  page  of  List  of  Objectives 
(Attachment  2). 

3.  Make  any  conaents  regarding  definitions  on  the  list  of  definitions 
provided . 

A.  Return  the  response  sheets  and  list  of  definitions  using  the  pre-addressed 
return  envelope  no  later  than  26  March  1991. 


Note:  The  objectives  developed  are  meant  to  describe  the  current  training 
course  as  identified  by  the  training  syllabus  and  CAT  leader  notebook.  Some 
objectives  may  only  relate  to  amterial  that  is  never  openly  discussed  in  the 
course  but  is  provided  in  the  book.  Specific  recommendations  for  improvements 
or  additional  objectives  that  have  not  been  addressed  can  be  identified  on  the 
last  page  of  the  list  of  objectives. 
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Identify  the  extent  to  vrtiich  you  cover  each  of  the  objectives  below  in 
your  presentations  of  a  particular  training  block. 


(1)  I  cannpletely  cover  this  objective 

(2)  I  partially  cover  this  objective 
(3}  I  do  not  cover  this  objective 

(4)  I  have  never  tau^t  this  section  of  the  course 


Block:  Overview  of  Total  Quality  Management 


1.  Identify  the  four  irajor  sub-systems  that  mke  vp  an 
organisations  TQ  structure  from  a  TQ  structure  chart. 

(1) 

(2) 

(3) 

(4) 

2.  Identify  the  primary  role  of  the  four  major  s\d>-systems  in  an 
orgamzations  TQ  structure. 

(1) 

(2) 

(3) 

(4) 

3.  Identify  the  morbership  of  the  four  najor  sub-systens  that 
mke  up  an  organizations  TQ  structure  from  a  list  of  options. 

(1) 

(2) 

(3) 

(4) 

4.  Identify  the  expected  life  of  the  four  mjor  sub-systems  that 
mke  up  an  organizations  TQ  structure  from  a  list  of  options. 

(1) 

(2) 

(3) 

(4) 

Block:  Values 

S.  Have  an  appreciation  for  the  importance  of  valiies  in  day  to 
day  interactions  that  relate  to  both  work  and  social  environment. 

(1) 

(2) 

(3) 

(4) 

6.  Have  an  appreciation  for  the  difference  between  maintenance 
factors  and  motivational  factors  identified  by  Herzberg. 

(1) 

(2) 

(3) 

(4) 

7.  Have  an  appreciation  for  «diat  the  Strength  Deployment 

Inventory  (SDI)  describes  and  depicts. 

(1) 

(2) 

(3) 

(4) 

8.  Identify  that  the  Strength  Deployment  Inventory  (SDI)  is 
based  on  personal  inputs. 

(1) 

(2) 

(3) 

(4) 

Block:  Group  Dynamics 

9.  Identify  the  5  stages  in  Tuckran's  model  for  group 
development. 

(1) 

(2) 

(3) 

(4) 

10.  Identify  the  characteristics  of  each  stage  in  TucXmn's  model 
for  group  development. 

(1) 

(2) 

(3) 

(4) 

11.  Identify  the  fmctions  of  a  leader. 

(1) 

(2) 

(3) 

(4) 

12.  Have  an  a^reciation  for  what  does  and  does  not  constitute  a 

(1) 

(2) 

(3) 

(4) 

leader. 
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Identify  the  extent  to  idiich  you  cover  each  of  the  objectives  below  in 
your  presentations  of  a  particular  training  block. 

(1)  I  conpletely  cover  this  objective 

(2)  I  partially  cover  this  objective 

(3)  I  do  not  cover  this  objective 

(4)  I  have  never  taught  this  section  of  the  course 


13.  Have  an  appreciation  for  team  characteristics. 

14.  Have  an  appreciation  for  on-going  team  assessment. 

15.  Have  an  appreciation  for  goals  of  work  teams. 

16.  Have  an  appreciation  for  team  norms. 

17.  Have  an  appreciation  for  problems  that  teams  ray  encounter. 

18.  Have  an  appreciation  for  behaviors  that  indicate  a 
motivational  problem  in  a  team. 

19.  Have  an  appreciation  for  characteristics  of  effective  teams. 

20.  Have  an  appreciation  for  aids  for  creating  a  conducive 
environment  for  team  effectiveness. 

21.  Have  an  appreciation  for  the  need  and  timing  of  consensus 
decision  making. 

22.  Have  an  appreciation  for  the, sole  of  constructive  conflict  in 

discussions.  '** 

23.  Have  an  appreciation  for  the  strategies  for  managing 
difficult  people  developed  by  Hanning  and  Haddock. 

24.  Identify  some  barriers  to  effective  comrunication. 

25.  Identify  the  responsibilities  of  the  speaker  and  listener  in 
a  conversation. 

26.  Know  the  principles  for  giving  and  receiving  feedback. 

Blodt:  Prdalem  Solving 

27.  Have  an  appreciation  for  the  definition  and  principles  of 
problem  solving. 

28.  Identify  the  steps  in  the  problem  solving  process. 

29.  Have  an  appreciation  for  problem  statements,  objectives  and 
tasks  involved  in  problem  definition. 


(1) 

(2) 

(3) 

(4) 

(1) 

(2) 

(3) 

(4) 

(1) 

(2) 

(3) 

(4) 

(1) 

(2) 

(3) 

(4) 

(1) 

(2) 

(3) 

(4) 

(1) 

(2) 

(3) 

(4) 

(1) 

(2) 

(3) 

(4) 

(1) 

(2) 

(3) 

(4) 

(1) 

(2) 

(3) 

(4) 

(1) 

(2) 

(3) 

(4) 

(1) 

(2) 

(3) 

(4) 

(1) 

(2) 

(3) 

(4) 

(1) 

(2) 

(3) 

(4) 

(1) 

(2) 

(3) 

(4) 

(1) 

(2) 

(3) 

(4) 

(1) 

(2) 

(3) 

(4) 

(1) 

(2) 

(3) 

(4) 
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Identify  the  extent  to  idiich  you  cover  each  of  the  objectives  below  in 
your  presentations  of  a  particular  training  block. 

(1)  I  conf>letely  cover  this  objective 

(2)  1  partially  cover  this  objective 

(3)  I  do  not  cover  this  objective 

(4)  I  have  never  taught  this  section  of  the  course 


30.  Have  an  appreciation  for  the  tools  associated  with  problon 
definition. 

(1) 

(2) 

(3) 

(4) 

31.  Have  an  appreciation  for  tdiy  data  is  collected  and  the 
difficulties  often  involved. 

(1) 

(2) 

(3) 

(4) 

32.  Identify  the  differences  between  qualitative  and  quantitative 
data. 

(1) 

(2) 

(3) 

(4) 

33.  Have  an  appreciation  for  the  tasks  and  tools  associated  with 
gathering  infomation. 

(1) 

(2) 

(3) 

(4) 

34.  Have  an  appreciation  for  the  objectives  of  problon  analysis. 

(1) 

(2) 

(3) 

(4) 

35.  Have  an  appreciation  for  the  tasks  and  tools  that  can  be 
associated  with  problem  analysis. 

(1) 

(2) 

(3) 

(4) 

36.  Identify  problem  analysis  tools  as  they  relate  to 
quantitative  and  qualitative  data. 

(1) 

(2) 

(3) 

(4) 

37.  Identify  problem  analysis  tools  as  they  relate  to  data 
gathering  and  analysis. 

(1) 

(2) 

(3) 

(4) 

38.  Have  an  appreciation  for  the  objectives  in  evaluating 
alternate  solutions. 

(1) 

(2) 

(3) 

(4) 

39.  Have  an  appreciation  for  the  taslcs  and  tools  that  can  be 
associated  with  evaluating  alternate  solutions. 

(1) 

(2) 

(3) 

(4) 

40.  Have  an  appreciation  for  the  role  objectives  play  in  planning 
to  inplement  a  solution. 

(1) 

(2) 

(3) 

(4) 

41.  Have  an  appreciation  for  the  tasks  and  tools  that  can  be 
associated  with  planning  to  iirplenent  a  solution. 

(1) 

(2) 

(3) 

(4) 

42.  Have  an  appreciation  for  the  objective  and  concerns  in 
planning  to  iirplcncnt  a  solution. 

(1) 

(2) 

(3) 

(4) 

43.  Have  an  appreciation  for  the  tasks  and  tools  that  can  be 
associated  with  inplementing  a  solution. 

(1) 

(2) 

(3) 

(4) 
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Identify  the  extent  to  tdiich  you  cover  each  of  the  objectives  below  in 
your  presentations  of  a  particular  training  block. 


(1)  I  completely  cover  this  objective 

(2)  X  partially  cover  this  objective 

(3)  I  do  not  cover  this  objective 

(4)  I  have  never  tau^t  this  section  of  the  course 


Block:  Tools  and  Te<d»u.qtues 


44.  Identify  the  operational  definition  of  brainstorming. 

(1) 

(2) 

(3) 

(4) 

45.  Identify  the  operational  definition  of  Nominal  Group 

Technique  (NCTT). 

(1) 

(2) 

(3) 

(4) 

46.  Identify  the  operational  definition  of  Cause  and  Effect 
Analysis. 

(1) 

(2) 

(3) 

(4) 

47.  Identify  the  operatiodnal  definition  of  Check  Sheets. 

(1) 

(2) 

(3) 

(4) 

48.  Identify  the  operational  definition  of  Pareto  Analysis. 

(1) 

(2) 

(3) 

(4) 

49.  Identify  the  operational  definition  of  Flow  Charting. 

(1) 

(2) 

(3) 

(4) 

50.  Identify  the  operational  definition  of  Solution  Selection 
Matrix. 

(1) 

(2) 

(3) 

(4) 

51.  Identify  the  operational  definition  of  Force  Field  Analysis. 

(1) 

(2) 

(3) 

(4) 

52.  Identify  the  operational  definition  of  Action  Planning. 

(1) 

(2) 

(3) 

(4) 

53.  Identify  how  a  specific  tool  or  technique  may  be  used  in  the 
problem  solving  process. 

(1) 

(2) 

(3) 

(4) 

54.  Identify  a  tool  or  technique  when  presented  with  the  steps  of 
that  tool  or  technique. 

(1) 

(2) 

(3) 

(4) 

55.  Identify  a  diagram  in  the  CAT  leader  notebodt  «duch 
represents  a  Cause  and  Effect  diagram. 

(1) 

(2) 

(3) 

(4) 

56.  Identify  a  diagram  in  the  CAT  leader  notebook  idiich 
represents  a  flow  cdiart. 

(1) 

(2) 

(3) 

(4) 

57.  Identify  a  diagram  in  the  OVT  leader  notebodt  which 
represents  a  Source  Selection  Matrix. 

(1) 

(2) 

(3) 

(4) 

58.  Identify  a  diagram  in  the  CAT  leader  notebook  «duch 
represents  a  Pareto  diart. 

(1) 

(2) 

(3) 

(4) 

59.  Identify  a  diagram  in  the  CAT  leader  notebodc  idiich 
represents  a  Check  Sheet. 

(1) 

(2) 

(3) 

(4) 
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Id«ntify  the  extent  to  which  you  cover  each  of  the  objectives  below  in 
your  presentations  of  a  particular  training  block. 


(1)  I  conpletely  cover  this  objective 

(2)  I  partially  cover  this  obj^tive 

(3)  I  do  not  cover  this  objective 

(4)  1  have  never  taught  this  section  of  the  course 


60.  Identify  the  diagram  in  the  CAT  leader  notebook  whicdi 
represents  a  Force  Field  Diagram. 

(1) 

(2) 

(3) 

(4) 

Block:  Meeting  Effectiveness 

61.  Have  an  appreciation  for  the  reguironents  of  holding  an 
effective  meeting. 

(1) 

(2) 

(3) 

(4) 

62.  Have  an  appreciation  for  the  requirements  in  making  an 

(1) 

(2) 

(3) 

(4) 

effective  nanagement  presentation. 


nus  concludes  the  objectives  in  Section  1 


Section  2 


Please  circle  all 
time  constraints. 


ex.  (64.1  Have  an 


objectives  in  Section  1  that  are  not  always  addressed  solely  due  to 
appreciation  for. . . 


Section  3 

If  you  responded  with  a  3  or  4  for  any  objective,  please  briefly  indicate  the  rationale 
for  that  response  by  that  objective  in  section  1  (i.e.  not  relevant,  uncomfortable  with 
topic,  never  enough  time,  etc.). 


Section  4 

Are  there  any  connents  regarding  these  objectives  or  are  there  subjects  that  were  missed? 
If  so,  please  elaborate  (use  back  if  necessary). 
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Attachaent  4: 


List  of  Definitions 


Brainstormina  -  Verbal  (eneration  of  ideas  in  a  group  that  they  key  specific 
details  of  a  topic  through  encouraging  spontaneous,  rapid  and  creative  input. 
Judgement  is  deferred. 

Nominal  Group  Technique  fNGTl  -  A  method  of  reaching  consensus  that  is  a 
refinement  of  brainstorming.  It  involves  the  individual  written  recording  of 
ideas  followed  by  a  group  discussion  that  ranks  and  prioritizes. 

Cause  and  Effect  -A  graphical  refinement  of  brainstorming  designed  to 

help  identify  the  root  causes  of  a  problem.  Individuals  propose  causes  to  a 
problem  that  are  graphically  represented  as  branches  or  twigs  that  feed  into 
the  problem. 

Checksheets  -  A  method  of  accumulating  and  documenting  data  to  assist  in  the 
detection  of  patterns  or  frequency  of  occurrence  of  events. 

Pareto  Anslvsia  ••  A  prioritization  tool  involving  a  graphical  technique  where 
data  is  arranged  in  descending  order  to  rank  the  contributors  to  a  problem  by 
their  order  of  importance. 

Plow  Charting  -  A  pictorial  representation  of  all  major  sequential  steps  in  a 
process . 

Solution  Selection  Matrix  -  A  numerically  comparative  chart  that  rates  each 
proposed  solution  with  respect  to  solution  selection  criteria.  The  sum  of 
ratings  for  each  solution  against  the  criteria  provides  a  structured  method 
for  comparison  of  solutions. 

Force  Field  Analysis  -  A  graphical  technique  that  identifies  the  driving  and 
restraining  factors  that  contribute  to  the  current  status  of  a  problem.  A 
vertical  line  represents  the  current  situation  and  drivers  and  restrainers  are 
opposed  to  each  other  on  opposites  sides  of  the  line  to  represent  the  reason 
for  the  current  status. 

Action  Planning  -  A  written  plan  of  action  for  accomplishing  a  stated  goal. 

It  includes  specific  objectives,  required  aiajor  activities,  personnel 
assignments  and  responsibilities,  and  an  established  schedule  for 
accomplishment . 
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Appendix  C;  Content  Validity  Ratio  Values  for  Objectives 
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1.00  0.78  X 

0.78  0.78  X  Content  Validity  Index  (CVI)  =  0.73 

1.00  0.78  X 

1.00  0.78  X  X  =  taugbt  by  all  instructors  (p  <  .05) 

1.00  0.78  X 


Appendix  D;  CAT  Leader  Training  Knowledge  Test 


Corrective  Action  Teem  (CAT)  Leeder  Traioins 

Eva lua t ion 


CAT  leader  training  ia  currently  receiving  a  major  evaluation  to  introduce 
iaprovoaenta  into  the  course.  You  are  one  of  three  classes  to  support  in 
class  aeasurenenta . 

This  portion  of  the  evaluation  of  CAT  leader  training  ia  designed  to  identify 
the  knowledge  of  course  aaterial  by  participants  both  before  and  after 
training.  You  do  not  need  to  identify  yourself  since  only  group  statistics 
will  be  used. 

Results  should  provide  a  foundation  for: 

-tailoring  the  TQ  overview  where  staterial  is  already  understood, 
-investigating  the  isipact  of  prior  esperience,  training,  end  education 
on  course  swterial. 

-identifying  areas  of  strength  to  tailor  discussions. 

-deleting  redundant  staterial. 

-esiphasizing  the  proper  topics. 

The  test  is  divided  into  two  parts.  Section  1  addresses  demographic  issues 
while  section  2  relates  to  your  knowledge  of  course  staterial.  This  package 
covers  store  inforstation  than  what  ia  presented  during  training  to  measure 
topics  that  stay  be  inclttded  in  the  future. 

Pleaa«  follow  ^ha  iostructiona  below: 

1.  Note  your  start  time  at  the  bottoa  of  the  page. 

2.  Using  e  number  2  pencil,  mark  the  response  sheet  (AFIT  Form  IID)  to  answer 
the  questions  in  the  package  provided. 

Answer  all  questions  (unless  otherwise  directed) 

3.  Note  your  finish  time  at  the  bottoa  of  this  page. 


START  TIME: 

FINISH  TIME: 


THANK  YOU 
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section  1:  P«r«onal/orgaiiiz*tional 


If  Military,  what  is  your 

current 

rank  (leave  blank  if  civilian)? 

A. 

NCO  (E-1  to  E-5) 

D.  Captain 

B. 

NCO  (B-6  to  B-9) 

E.  Major 

C. 

Lieutenant 

P.  Lt  Col/Col 

If  civilian,  what  ia  your 

current 

grade  (leave  blank  if  in  military)? 

A. 

GS-3  thru  GS-S 

E. 

GM-13  thru  GH-14 

B. 

GS-4  thru  GS-8 

P. 

GM-15 

C. 

GS-9  thru  GS-11 

G. 

Other 

D. 

GS-12  thru  GS-13 

What 

was  your  age  at  your 

last  birthday? 

A. 

Under  21 

E. 

36-40  I.  56  or  older 

B. 

21-25 

P. 

40-45 

C. 

26-30 

C. 

46-50 

D. 

31-35 

H. 

51-55 

4.  What  ia  your  gandar? 

A.  Faaale  B.  Hale 

5.  What  ia  rour  highest  achieved  education  level? 

A.  High  School  Diploma 

B.  Associate's  Degree 

C.  Bachelor's  Degree 

D.  Master's  Degree 

E.  Doctoral  Degree 

4.  Are  you  currently  leading  a  Corrective  Action  Team  (CAT)? 

A.  Yes 

B.  No 

C.  I  am  an  assistant  leader 

7.  How  many  CATs  or  Critical  Process  Teams  (CPTs)  have  you  been  involved  with  (as  a 
leader  or  participant)? 

A.  0 

B.  1 

C.  2 

D.  3  or  more 

B.  Do  you  currently  have  a  CAT  topic  assigned  to  you  by  your  organization? 

(Note:  Leave  blank  if  you  are  not  planning  to  hinction  as  a  CAT  leader.) 

A.  Yes 

B.  No 

C.  No,  but  one  will  be  assigned  upon  my  iamediate  return  from  training. 
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Hav*  you  raceivad  any  of  tha  following  training/aducation  ralatad  to  group  problaa-aolving 
and  taaa  dynaaica? 


9. 

Initial  TQ  Team  Training  with  Cumberland 

A.  Tea 

B. 

Mo 

10. 

Transition  TQ  Team  Training  with  ASO/Ciabarland 

A.  Tes 

B. 

Mo 

11. 

Critical  Process  Tesm  (CPT)  Training  with  ASD/Cumbarland 

A.  Tes 

B. 

Mo 

12. 

Quality  Leadership  Workshop  training  with  Cumberland 

A.  Yes 

B. 

Mo 

13. 

APIT  -  Quality  Circle  Facilitator /Leader  Course  (1979-1990) 

A.  Tes 

B. 

No 

14. 

APLC  -  Process  Action  Team  (PAT)  Training 

A.  Yes 

B. 

No 

IS. 

ALC  -  Quality  Circle  Facilitators  Training 

A.  Yes 

B. 

No 

16. 

Other  (nlease  snecify) 

A.  Yes 

B. 

No 

Uaa  tha  following  rating  tcala  for  tha  nazt  5  atataaantw  to  azpraaa  your  own  faelinga  on 
your  ability  to  parfora  varioua  taaka. 

A.  Meant  you  atronaly  diaanraa  with  tha  stataaont 

B.  Means  you  aioderatelT  disagree  with  the  stateaent 

C.  Means  you  slightly  disagree  with  the  statement 

D.  Means  you  neither  disagree  nor  agree  with  the  stateteent 

E.  Means  you  slightly  agree  with  the  stateaent 
P.  Means  you  moderately  agree  with  the  stateawnt 
G.  Means  you  strongly  agree  with  the  stateaent 

17.  I  faal  I  have  the  capability  to  lead  a  group  of  individuals  through  the  problea- 
solving  process  to  resolve  an  organizational  issue. 

18.  I  feel  I  have  tha  capability  to  explain  the  TQ  structure  and  philosophy  to 
individuals  in  ay  organization. 

19.  1  feel  comfortable  using  the  tools  and  techniques  of  problem-solving  in  a  group 
environment  to  address  an  organizational  issue. 

20.  1  feel  I  have  the  capability  to  prepare  and  facilitate  an  effective  meeting. 

21.  I  faal  I  understand  tha  subject  of  group  dynamics  enough  to  build  an  effective  team 
and  solve  an  organizational  issue. 
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Knowledge  Evaluation 


In  the  following  diagram,  match  the  major  sub-systeois  of  a  "generic"  Total  Quality  (TQ) 
atructure  in  an  ASD  organization. 


22.  _  Critical  Proceaa  Team 

23.  _  TQ  Team 

24.  _  Corrective  Action  Team 

25.  _  Executive  Steering  Committee 


Correctly  match  the  TQ  sub-system  with  its  role: 


Sub-System 


Bsls. 


26.  _  Critical  Process  Team  (CPT)  A.  Design  and  implement  a  system  to 

continuously  improve  a  designated  high-level 
cross  divisional/  functional  process. 

27.  _  TQ  Team 

B.  Provide  overall  planning  coordination  and 
guidance  of  the  iatproveakent  process .  Monitor 

28.  _  Corrective  Action  Team  (CAT)  executive  action  plans  and  coordinate  labor/ 

management  involvement. 


29.  _  Executive  Steering  Coaxaittee  C.  Design,  implement,  and  manage  a  system  to 

cause  improveaients  in  an  organization. 


D.  Address  a  specific  problem  in  an 
organization  and  make  recommendations  to 
management  for  action. 


Correctly  match  the  TQ  subsystem  with  its  aiembership: 

Subsystem  Me 


Critical  Process  Team 


A.  Cross  functional,  high-level  executives. 


31.  _  TQ  Team 

32.  _  Corrective  Action  Team  (CAT) 


B.  An  entire  specially  trained  team  with  owners 
of  various  elements  of  a  process,  plus  others 
•dio  would  be  helpful. 


33.  _  Executive  Steering  Conaittee  C.  Cross  section  of  organizational  personnel 

with  varying  ranks  and  grades. 


0.  Individuals  idio  are  knowledgeable  of  the 
problem.  Coordination  is  by  a  specially  trained 
leader. 
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Correctly  classify  each  TQ  sub~system  as  one  of  the  following  structures  in  an 
organization: 


34.  _  Critical  Process  Team 

35.  _  TQ  Team 

36.  Corrective  Action  Team 

37.  _  Exec  Steering  Coasittae 


A.  Permanent  Organization 

A.  Permanent  Organization 
A.  Permanent  Organization 
A.  Permanent  Organization 


B.  Temporary  Organization 

B.  Temporary  Organization 
B.  Temporary  Organization 
B.  Temporary  Organization 


38.  The  Strength  Deployment  Inventory  is  best  described  by  the  following: 

A.  A  tool  to  describe  observable  behavior  and  depict  general 
orientations/styles  based  on  an  individuals  self-perception. 

B.  A  iudeement  test  of  the  strengths  and  weaknesses  of  an  individual. 

C.  A  management  tool  for  matching  the  strength  of  an  individual  with 
possible  job  positions  to  fvin'Te  productivity. 

D.  A  eroup  tool  where  members  rate  each  other  on  a  strength/weakness  scale  to 
identify  possible  personality  conflicts  prior  to  undertaking  a  major  effort. 


Properly  order  the  5  stages  in  Tuckman's  siodel  of  group  development: 


39. 

_  Stage  I 

A. 

Performing 

40. 

_  Stage  II 

B. 

Koming 

41. 

_  Stage  III 

C. 

Adjourning 

42. 

_  Stage  IV 

D. 

Forming 

43. 

_  Stage  V 

E. 

Storming 

Correctly 

identify  the  characteristics 

of  the 

individual  stages  of  Tuckman's 

identified 

below: 

Stage 

Characterized  bv: 

44. 

_  Performing 

A. 

Testing  and  Dependence 

45. 

_  Norming 

B. 

Attempts  at  closure  and  ending 

46. 

_  Adjotiming 

the  team 

47. 

_  Forming 

C. 

Development  of  team  cohesion 

48. 

_  Storming 

D. 

Intra-team  conflict 

E. 

Functional  Role  relatedness 

49.  Which  of  the  following  is  not  a  function  of  a  leader: 

A.  Guides  a  team  in  delineating  a  vision  and  setting  gosls. 

B.  Mobilizes  support  and  resources  for  gosl  achievement  (helps  translate  goals  into 
actions) 

C.  Manases  the  interaction  among  mesibers  so  that  the  energies  of  the  team  remain 
focused  on  problem  solving. 

D.  Provides  a  charismatic  role  model  that  encourages  resolving  group 
conflict  and  creating  an  atmosphere  to  meet  connon  objectives. 
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50.  Which  of  the  following  is  not  s  barrier  to  effective  eoMinication: 

A.  Expectations 

B.  Assueptions 

C.  The  receiver  is  a  supervisor 

D.  The  receiver  mmy  not  be  listening 

E.  The  receiver  any  not  understand 

Out  of  the  following  list,  identify  the  first  3  steps  of  the  6  steps  of  problem  solving 
(order  not  iaportsnt): 


51.  _ 

A.  Establish  the  desired  goal 

52.  _ 

B.  Docunmnt  the  specifics 

53.  _ 

C.  Define  the  problem 

D.  DocusMnt  a  plan  of  action 

E.  Gather  information 

F.  Analyze  the  problem 

Out  of  the  following  list,  identify  the  last  three  steps  of  the  6  steps  of  problem  solving 
(order  not  important): 


54,  _ 

A. 

Evaluate  alternate  solutions 

55.  _ 

B. 

Obtain  managesmnt  buy~in 

56.  _ 

C. 

Plan  the  inpleawntation 

D. 

Implesient  the  solution 

E. 

List  solution  options 

F. 

Docusient  a  plan  of  action 

57.  Which  one  of  the  following  lists  of  tools  and  techniques  is  associated  with  the 
collection  of  qualitative  data; 

A.  Check  Sheets  and  questionnaires 

B.  Graphs  histograsis  and  Pareto  charts 

C.  Brainstorming,  Interviews  and  Force  Field  Analysis 

56.  Which  one  of  the  following  lists  of  tools  and  techniques  is  associated  with  the 
collection  of  quantitative  data: 

A.  Graphs  and  histograms 

B.  Brainstorming  and  interviews 

C.  Nominal  Group  Technique 

59.  Out  of  the  following  list,  identify  that  item  that  is  not  considered  quantitative 
data: 

A.  The  tiaw  to  coordinate  a  letter 

B.  The  types  of  priorities  in  an  action  plan 

C.  The  nusiber  of  clerks  in  an  office 

D.  The  total  cost  of  a  weapon  system 

60.  Out  of  the  following  list,  identify  the  item  that  is  no^  considered  qualitative  data: 

A.  The  nasws  of  each  member  of  a  team 

B.  The  approval  steps  in  signing  out  a  letter 

C.  The  factors  chat  are  considered  in  a  promotion 

D.  The  aanunt  of  a  raise 


* 
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Match  each  specific  tool  or  technique  with  ita  corresponding  operational  definition: 


61.  _  Cause  and  Effect  A.  A  pictorial  representation  of  all  sujor  sequential 

Oiagraas  steps  in  a  process. 

B.  A  prioritization  tool  involving  a  graphical  technique 
where  data  is  arranged  in  descending  order  to  rank  the 

62.  _  Force  Field  Analysis  contributors  to  a  problem  by  their  order  of 

importance. 

A  written  plan  of  action  for  accomplishing  a  stated 
goal.  It  includes  specific  objectives,  required  major 
activities,  personnel  assignments  and 
responsibilities,  and  an  established  schedule  for 
accomplishment . 

D.  Verbal  generation  of  ideas  in  a  group  that  they  key 

64.  _  Nominal  Group  apecific  details  of  a  topic  through  encouraging 

Technique  (NGT)  spontaneous,  rapid  and  creative  input.  Judgement  is 

deferred. 

E.  A  Biethod  of  accumulating  and  documenting  data  to 

65.  _  Plow  Chart] -'g  assist  in  the  detection  of  patterns  or  frequency  of 

occurrence  of  events. 

A  graphical  technique  that  identifies  the  driving  and 
restraining  factors  that  contribute  to  the  current 
status  of  a  problem.  A  vertical  line  represents  the 
current  situation  and  drivers  and  restrainers  are 
opposed  to  each  other  on  opposites  sides  of  the  line 
to  represent  the  reason  for  the  current  status. 

67.  _  Action  Planning  C.  A  graphical  refinesient  of  brainstorming  designed  to 

help  identify  the  root  causes  of  a  problem. 

Individuals  propose  causes  to  a  problem  that  are 
graphically  represented  as  branches  or  twigs  that  feed 
into  the  problem. 

68.  _  Pareto  Analysis 

H.  A  numerically  coiq>arative  chart  that  rates  each 
proposed  solution  with  respect  to  solution  selection 
criteria.  The  sum  of  rstings  for  each  solution 
against  the  criteria  provides  a  structured  method  for 

69.  _  Brainstorming  comparison  of  solutions. 

I.  A  method  of  resching  consensus  that  is  refinement  of 
brainstorsdng.  It  involves  the  individual  written 
recording  of  ideas  followe<1  by  a  group  discussion  that 
ranks  and  prioritizes. 


P. 

66.  _  Solution  Selection 

Hatriz 
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70.  Identify  the  tool  or  technique  that  ie  not  aDDronriate  for  defining  a  problem: 

A.  Brainatorming 

B.  Cause  and  Effect  Diagrams 

C.  Plow  Charting 

D.  Force  Field  Analysis 

71.  Identify  the  tool  or  technique  that  in  not  appropriate  for  gathering  information: 

A.  Solution  Selection  Matrix 

B.  Check  Sheata 

C.  Pareto  Analysis 

D.  Cause  and  Effect  Diagrams 

72.  Identify  the  Cool  or  technique  that  is  not  appropriate  for  analyzing  a  problem: 

A.  Flow  Charting 

B.  Nominal  Group  Technique  (NGT) 

C.  Action  Planning 

D.  Pareto  Analysis 

73.  Identify  the  tool  or  technique  that  it  not  aoprooriate  for  planning  to  implement  a 
solution: 

A.  Brainstorming 

B.  Force  Field  Analysis 

C.  Flow  Charting 

D.  Check  Sheets 


In  the  next  five  questions,  given  the  steps  in  the  process,  identify  the  tool  or  technique. 
Key  concepts  for  each  tool  or  technique  are  highlighted. 


74. 

The  steps  on  the 

right  describe 

which  tool  or  technique? 

A. 

Cause  and  Effect 

Step 

1 

Identify  exactly  ««hat  event  is  to  be  observed 

B. 

Check  Sheets 

Step 

2 

Decide  on  the  time  period  for  data 

C. 

Nominal  Group 

Step 

3 

Design  a  form  that  is  clear  and  easy  to  use.  Clearly 

Technique  (NGT) 

label  columns  and  leave  enough  space  to  enter  data. 

D. 

Flow  Charting 

Step 

4 

Collect  data  consistentlv  and  honestly 

Step 

5 

Total  results 

75. 

The  steps  to  the 

right  describe 

which  tool  or  technique? 

A. 

Pareto  Analysis 

Step 

1 

Determine  how  to  collect  and  classify  your  data. 

B. 

Brainstorming 

Step 

2 

Collect  data  for  a  specified  period  of  time. 

C. 

Check  Sheets 

Step 

3 

SuBDsrite  recorded  data  in  a  table.  Arranee  from 

D. 

Cause  and  Effect 

large  to  small  frequencies. 

Diagrams 

Step 

4 

Set  up  chart.  Category=Horizontal  axis 

<)uantity=vertical  axis 

Step 

5 

Plot  the  data  in  dcaeandinx  order  with  highest 

value  category  on  the  far  left . 

Step 

6 

Draw  bars. 

Step 

7 

Plot  Cum  line  (if  used)  to  scale 
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76.  The  steps  to  the  right  describe  which  tool  or  technique? 


A.  Plow  Charting 

B.  Nominal  Croup 

Technique  (NOT) 

C.  Force  Field  Analysis 

D.  Brainstorming 


Step  1  Appoint  a  recorder 

Step  2  Hake  a  list  of  all  itesis 

Step  3  Generate  as  mane  ideas  as  possible 

Step  4  Encourage  freewheeling 

Step  5  No  criticism  or  discussion  allowed 

Step  6  All  siembers  have  an  equal  opportunity 

Step  7  Rapid  Biovement  around  the  group 

Step  8  Let  ideas  incubate 


77.  The  steps  to  the  right  describe  which  tool  or  technique? 


Cause  and  Effect 
Diagram 
Nominal  Croup 
Technique  (NGT) 
Brainstorming 
Force  Field  Analysis 


Step  1  Select  an  undesturbable  meeting  place 
Step  2  Prepare  question  and  post  on  flip  chart 
Step  3  Participants  seouentiallv  share  ideas,  one  at  a  tiaie, 
with  no  discussion. 

Step  4  Record  all  ideas  openly  and  visibly  exactly  as 
suggested 

Step  S  Continue  process  until  no  store  ideas  are  generated 
Step  6  Conduct  a  brief  discussion  only  to  clarify  each  idea 
Step  7  Participants  silently  choose  their  top  5-8  ideas 
and  rank  order  them 

Step  8  Record  the  choices  on  3XS  cards  or  post-its 
Step  9  Repeat  until  the  top  5  items  are  agreed  to  by  all 
participants 


78.  The  steps  to  the  right  describe  which  tool  or  technique? 


Force  Field  Analysis 
Cause  and  Effect 
Diagrasis 
Check  Sheets 
Solution  Selection 
Hatrix 


Step  1  Identify  problem  and/or  solution 
Step  2  Identify  the  drivins  factors  which  result  if  the 
change  is  activated 

Step  3  Identify  the  restraining  factors  that  tend  to  kee 


Step  3  Identify  the  restraining  factors  that  tend  to  keep 
the  change  from  occurring 
Step  4  Estimate  strength  of  each 

Step  S  Identify  each  as  Type  I,  Type  II,  or  Type  III  issue 

Step  6  Develop  Strategy  to  impleaient  the  change 

Maximize  driving  forces 
Minimize  restraining  forces 
Step  7  Change  restrainer  to  a  driver 

Step  8  Sunxnarizc  results 


79.  This  diagram  is  associated  with  which  of  the  following  tool  or  technique? 


A.  Cause  and  Effect  Diagrams 

B.  Pareto  Analysis 

C.  Action  Plsnning 

D.  Check  Sheets 


143 


80.  This  diagram  is  associated  with  which  of  the  following  tool  or  technique? 


81.  This  diagram  is  associated  with  which  of  the  following  tool  or  technique? 

A.  Force  Field  Analysis 

B.  Cause  and  Effect  Diagrams 

C.  Action  Planning 

D.  Flow  Charting 


82.  This  diagram  is  associated  with  which  of  the  following  tool  or  technique? 


« 
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83.  Thia  diasraa  ia  aaaoeiatad  with  which  of  the  following  tool  or  technique? 


A.  Force  Field  Analyaia 

B.  Csuae  end  Effect  Oiacraaa 

C.  Check  Sheeta 

D.  Solution  Selection  Matrix 


STATUS  QUO  LWE 

DRIVERS  RESTRAINERS 


I 


a 


84.  Thia  diagraa  ia  aaaociatad  with  which  of  the  following  tool  or  technique? 


A.  Flow  Charting 

B.  Cauae  and  Effect  Diagraae 

C.  Force  Field  Analyaia 

D.  Action  Planning 
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Appendix  E:  CAT  Leader  Graduate  Questionnaire 
DEPARTMENT  OF  THE  AIR  FORCE 

MKADOgAHTtRS  AEKOMAUTICAl.  STSTEMS  IMVtSIOM  (AFSC) 

WmOHT-rATTCREOM  AIR  AORCE  RASE.  OHIO  AMJMM* 


30  April  1991 

KT  Corrective  Action  Team  (CAT)  Leader  Training  Effectiveness 
Research 

n  ASD  CAT  Leader  Training  Participant 

1.  This  package  is  part  of  a  coordinated  research  effort 
designed  to  evaluate  and  identify  metrics  for  Corrective  Action 
Team  (CAT)  leader  training.  Please  take  a  few  minutes  to 
complete  the  attached  questionnaire  and  return  it  in  the  enclosed 
envelope  by  14  June  1991.  The  questionnaire  was  specifically 
designed  to  take  less  than  20  minutes  to  complete. 

2.  As  individuals  who  have  received  the  training  and  had  the 
opportunity  to  apply  what  was  presented,  you  provide  a  unique 
source  of  information  that  can  guide  future  course  improvements 
to  make  CAT  leader  training  more  effective  in  the  ASD 
organization . 

3.  Your  responses  will  be  combined  with  those  from  other 
respondents  and  will  not  be  attributed  to  you  personally.  Any 
information  you  can  provide  through  feedback  and  suggestions  will 
be  useful.  Copies  of  the  final  results  for  the  entire  evaluation 
will  be  available  in  September  at  ASD/TQ.  If  you  have  any 
questions,  please  contact  Capt  Kirk  Streitmater,  AFIT/LSG,  255- 

Thank  you  for  your 


Atch 

Questionnaire 
AFIT  Form  IIC 
Definitions 
Return  Envelope 


4437  or  Mr  Dale  Missman,  ASD/TQ,  255-1755. 
consideration. 


Ronald  A.'  r 

oTrerton,  Colonel,  OSAF 

4 

Commander's 

Assistant 

1. 

for  Total 

Quality 

2. 

3. 

4. 

t 
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E*r‘±v«.cy  Act. 


In  accordance  with  paragraph  30,  APR  12-35,  the  following 
information  ia  provided  aa  required  by  the  privacy  act  of  1974: 

a.  Authority: 

(1)  5  D.8.C.  301,  Departmental  Regulations:  and 

(2)  10  O.8.C.  8012,  8ecretarv  of  the  Air  Poree. 

Eawqr a.  Duties.  Delegation  bv  Compensation:  and 

(3)  DOD  Instruction  1100.13,  17  Apr  88,  Surveys  of 
Department  of  Defense  Personnel :  and 

(4)  APR  30-23,  22  Sep  76,  Air  Porce  Personnel  Program. 

b.  Principle  purposes.  The  research  is  being  conducted  to 
collect  information  to  be  used  in  evaluating  the  corrective 
action  team  (CAT)  leader  training  conducted  at  Aeronautical 
System  Division  and  improve  the  Total  Quality  (TQ)  program 
effectiveness  in  that  organisation. 

c.  Routine  uses.  The  data  will  be  converted  to  information 
for  use  in  research  of  management  related  issues  dealing  with  TQ. 
Results  of  the  research,  based  on  data  provided,  will  be  included 
in  published  articles,  reports,  or  texts.  Distribution  of  the 
results  of  the  research,  based  on  the  data  collected,  whether  in 
written  form  or  presented  orally,  will  be  unlimited. 

d.  Participation  in  this  research  is  entirely  voluntary. 

e.  No  adverse  action  of  any  kind  may  be  taken  against  any 
individual  who  elects  not  to  participate. 


Irxs-tr-v»,c't.±ons 

1.  Review  each  item  and  answer  appropriately. 

-You  may  mark  on  the  package. 

-Refer  to  definitions  in  Attachment  3  (if  necessary). 

2.  Transfer  your  responses  onto  the  APIT  Porm  lie. 

-Do  not  identify  yourself  in  the  response  sheet. 

~Ose  a  number  2  lead  pencil  to  respond. 

3.  Place  the  response  sheet  in  the  envelope  provided. 
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Section  1:  Personal /organizational 


This  section  will  provide  general  backgroxmd  information  regarding 
characteristics  respondents  who  have  taken  CAT  leader  training. 


If 

military,  what  is 

VQur  current 

rank?  (leave  blank  if 

civilian) 

1. 

Enlisted  (E-1  to 

E-5) 

4.  Captain 

2. 

Enlisted  (E-6  to 

E-9) 

5.  Major 

3. 

Lieutenant 

6.  Lt  Ccl/Colonel 

If 

civilian,  what  is 

your  current 

grade?  (leave  blank  if 

military) 

1. 

GS-3  thru  GS-5 

5.  GS-13 

thru  GM-14 

2. 

GS-6  thru  GS-8 

6.  GM-15 

3. 

GS-9  thru  GS-10 

7 .  Other 

4. 

GS-11  thru  GS-12 

3.  What  was  your  age  at 

1.  Under  21 

2.  21-25 

3.  26-30 

i.  31-35 

4.  What  is  your  sex? 

1.  Female  2.  Male 

5.  What  is  your  highest  achieved  education  level? 

1.  High  school  Diploma  4.  Master’s  Degree 

2.  Associate's  Degree  5.  Doctoral  Degree 

3.  Bachelor's  Degree 

Section  2:  Training  Information 

This  section  will  provide  information  that  will  allow  time  trend  analysis 
for  CAT  leader  training  applications. 

6.  Are  you  currently  in  the  organization  that  sponsored  you  for  CAT 
leader  training? 

1.  yes  2.  Ho 


7 . 

Select 

training 

date 

(Note : 

training 

date 

is 

on 

your  mailing  label) 

1 . 

1 

Sep  89 

4. 

17  Hov  89 

7. 

9  Feb  90 

2. 

29 

Sep  89 

5. 

8  Dec  89 

8. 

Other  -  item  8  or 

9 

3. 

13 

Oct  89 

6. 

12  Jan  90 

8. 

Sel ect 

training 

date 

(Note: 

training 

date 

is 

on 

your  mailing  label) 

1. 

9 

Mar  90 

4. 

8  Jun  90 

7  . 

10  Aug  90 

2. 

6 

Apr  90 

5. 

13  Jul  90 

8. 

Other  -  item  7  or 

9 

3. 

4 

May  90 

6. 

20  Jul  90 

9. 

Select 

training 

date 

(Note: 

training 

date 

is 

on 

your  mailing  label) 

1. 

14 

Sep  90 

4. 

7  Dec  90 

7. 

12  Apr  90 

2. 

12 

Oct  90 

5. 

8  Feb  90 

8. 

Other  -  item  7  or 

8 

3. 

16 

Hov  90 

6. 

22  Mar  90 

your  last  birthday? 

5.  36-40 

6.  41-45 

7.  46-50 

8.  51-55 


56  or  older 
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Section  3:  Assignments  end  status  of  CATs 

This  section  will  provide  information  on  CAT  involvement  and  status. 

10.  How  many  CATs  have  you  been  involved  with  (as  a  leader  or 
participant}? 

1.  0  (SKIP  to  item  21) 

2.  1 

3.  2 

4.  3  or  more 

11.  Which  of  the  following  best  describes  the  status  of  your  most  recent 
CAT? 

1.  CAT  currently  open. 

2.  Recommendations  made  and  implemented  and  CAT  disbanded. 

3.  Recommendations  made  but  not  yet  implemented  and  CAT  disbanded. 

4.  CAT  approach  found  inappropriate  and  other  approaches  pursued  and 
CAT  disbanded. 

5.  Unknown  -  I  am  no  longer  in  that  organization. 

12.  Regarding  your  most  recent  CAT  affiliation,  how  would  you  rate  the 
applicability  of  using  a  CAT  problem-solving  approach  in  addressing 
that  specific  issue? 

1.  The  CAT  problem-solving  approach  is/was  very  applicable. 

2.  The  CAT  problem-solving  approach  is/was  somewhat  applicable. 

3.  The  CAT  problem-solving  approach  is/was  not  very  applicable. 

4.  The  CAT  problem-solving  approach  is/was  not  applicable. 

13.  Have  you  served/are  you  serving  as  a  CAT  leader  since  receiving 
training? 

1.  Tes 

2.  No  -  Not  yet  assigned  an  issue,  but  planned  (SKIP  to  item  15). 

3.  No  -  No  plans  to  function  as  a  CAT  leader  (SKIP  to  item  15). 

14.  what  was  the  time  gap  between  training  and  assignment  as  a  CAT  leader? 

1.  Was  assigned  CAT  issue  prior  to  training 

2.  0-1  month 

3.  2-3  months 

4.  4-6  months 

5.  7  months  -  1  year 

6.  Over  1  year 

15.  Are  you  currently  involved  in  a  CAT? 

(Give  highest  level  of  involvement  applicable) 

1.  No  (SKIP  to  item  21) 

2.  Tes,  I  am  leading  a  CAT. 

3.  Yes,  I  am  an  assistant  leader. 

4.  Yes,  I  am  a  participant,  but  not  a  leader. 

16.  Is  the  CAT  identified  in  item  15  the  first  that  you  have  been  involved 
with  since  training? 

1.  Tes  2.  No 


149 


17.  How  long  has  it  been  since  the  CAT  was  formed? 

1.  0-1  month 

2.  2-3  months 

3.  4-6  months 

4.  7  months  -  1 

5.  Over  1  year 


18.  Generally,  how  frequently  does  the  CAT  meet? 

1.  One  or  more  times  per  week. 

2.  At  least  every  other  week. 

3.  At  least  once  a  month. 

4.  As  the  need  arises. 

19.  How  long  has  it  been  since  the  last  meeting? 

1.  Less  than  a  week. 

2.  Less  the  two  weeks. 

3.  Less  than  a  month. 

4.  more  than  a  month. 


20.  How  often  is  the  CAT  status  briefed  to  the  organisational  leader? 

1.  At  least  every  week. 

2.  At  least  every  other  week. 

3.  At  least  once  a  month. 

4.  A  period  longer  than  a  month,  but  regularly  scheduled 

5.  As  the  need  arises. 


Section  4:  Perceived  abilities 

This  section  will  provide  information  of  perceived  abilities  to  perform 
different  tasks. 


Use  the  following  rating  scale  for  the  next  5  statements  to  express  your 
own  feelings  on  your  ability  to  perform  various  tasks. 

1.  Means  you  strongly  disagree  with  the  statement 

2.  Means  you  moderately  disagree  with  the  statement 

3.  Means  you  slightly  disagree  with  the  statement 

4.  Means  you  neither  disagree  or  agree  with  the  statement 

5.  Means  you  slightly  agree  with  the  statement 

6.  Means  you  moderately  agree  with  the  statement 

7.  Means  you  strongly  agree  with  the  statement 

21.  I  feel  I  have  the  capability  to  lead  a  group  of  individuals  through 
the  problem-solving  process  to  resolve  an  organisational  issue. 

22.  1  feel  I  have  the  capability  to  explain  the  TQ  structure  and 
philosophy  to  individuals  in  my  organisation. 

23.  I  feel  comfortable  using  the  tools  and  techniques  of  problem-solving 
in  a  group  environment  to  address  an  organisational  issue. 

24.  1  feel  I  have  the  capability  to  prepare  and  lead  an  effective  meeting. 

25.  I  feel  I  understand  the  subject  of  group  dynamics  enough  to  build  an 
effective  team  to  solve  an  organisational  issue. 
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Section  5:  Use  of  Tools  and  Techniques 


This  section  will  provide  information  on  the  level  *nd  frequency  of  use  of 
problem-solving  tools  and  techriques. 


Please  identify  the  highest  level  of  use  for  each  of  the  following  items. 
(l=lowest,  J=highest)  in  settings  other  than  training  courses. 

Note:  Definitions  for  each  item  are  provided  in  Attachment  3. 


Level  of  Dee 


1.  Have  not  used  personally  and  have  pot  seen  used. 

2.  Have  not  used  personally  but  have  seen  used  in  a  CAT  setting. 

3.  Have  not  used  personally  but  have  seen  used  in  an  organisation  setting 

4.  Have  personally  used  in  a  CAT  setting. 

5.  Have  personally  used  in  an  oroanisation  setting. 


26. 

Steps  in  problem-solving. 

(1) 

(2) 

(3) 

(4) 

(5) 

27. 

Brainstorming. 

(1) 

(2) 

(3) 

(4) 

(5) 

28. 

Nominal  Group  Technique  (NGT). 

(1) 

(2) 

(3) 

(4) 

(5) 

29. 

Cause  and  Effect  Analysis  (Fishbone 

Diagrams) .  (1) 

(2) 

(3) 

(4) 

(5) 

30. 

Pareto  Analysis. 

(1) 

(2) 

(3) 

(4) 

(5) 

31. 

Check  Sheets. 

(1) 

(2) 

(3) 

(4) 

(5) 

32. 

Flow  Charting. 

(1) 

(2) 

(3) 

(4) 

(5) 

33. 

Solution  Selection  Matrix. 

(1) 

(2) 

(3) 

(4) 

(5) 

34. 

Force  Field  Analysis. 

(1) 

(2) 

(3) 

(4) 

(5) 

35. 

Action  Planning. 

(1) 

(2) 

(3) 

(4) 

(5) 

Please  identify  the  number  of  times  you 

have  seen  this  tool  or 

technique 

used 

in  the  oast  three  years  outside  of 

training  courses 

(i  .e. 

in 

the 

work 

environment ) . 

Frequency  of  Use 


1.  Never 

2 .  Once 

3.  Twice 

4.  Three  or  more  times 


36. 

Steps  in  problem-solving. 

(1) 

(2) 

(3) 

(4) 

37. 

Brainstorming . 

(1) 

(2) 

(3) 

(4) 

38. 

Nominal  Group  Technique  (NGT). 

(1) 

(2) 

(3) 

(4) 

39. 

Cause  and  Effect  Analysis  (Fishbone  Diagrams). 

(1) 

(2) 

(3) 

(4) 

40. 

Pareto  Analysis. 

(1) 

(2) 

(3) 

(4) 

41. 

Check  Sheets. 

(1) 

(2) 

(3) 

(4) 

42. 

Flow  Charting. 

(1) 

(2) 

(3) 

(4) 

43. 

Solution  Selection  Matrix. 

(1) 

(2) 

(3) 

(4) 

44. 

Force  Field  Analysis. 

(1) 

(2) 

(3) 

(4) 

45. 

Action  Planning. 

(1) 

(2) 

(3) 

(4) 
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Please  identify  bow  you  would  rate  each  of  the  items  usefulness  as  a  tool 
for  team  problem-solving.  (Mote:  Definitions  are  in  Attaclunent  3} 

1.  Very  Oseful 

2.  Somewhat  Useful 

3.  Undecided 

4.  Rarely  Useful 

5.  Not  Useful 

46.  Steps  in  problem-solving. 

47.  Brainstorming. 

48.  Nominal  Group  Technique  (NGT). 

49.  Cause  and  Effect  Analysis  (Fishbone  Diagrams). 

50.  Pareto  Analysis. 

51.  Check  Sheets. 

52.  Flow  Charting. 

53.  Solution  Selection  Matrix. 

54.  Force  Field  Analysis. 

55.  Action  Planning. 

Section  6:  General  Comnents 

This  section  will  solicit  any  specific  feedback  regarding  areas  not  covered 
or  insufficiently  covered  by  previous  items. 

56.  After  having  taken  the  CAT  Leader  training  course  and  then  returned  to 
the  organizational  environment,  what  cotmnents  or  recommendations  do 
you  have  for  course  changes  and  improvements?  Explain  below 

Comments  on: 

TQ  Background  Section 

Personal  Values  Section 

Group  Dynamics  Section 

Steps  of  Problem  Solving  Section 

Tools  and  Techniques  Section 

Exercises 

Movies 

Final  Exercise 
Other 


THANK  YOU  FOR  YOUR  TIME 


(1)  (2)  (3)  (4)  (5) 

(1)  (2)  (3)  (4)  (5) 

ii!  B!  il!  !J!  Hi 

(1)  (2)  (3)  (4)  (5) 

(1)  (2)  (3)  (4)  (5) 

(1)  (2)  (3)  (4)  (5) 

(1)  (2)  (3)  (4)  (5) 

(1)  (2)  (3)  (4)  (5) 

(1)  (2)  (3)  (4)  (5) 
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Appendix  F;  CAT  Leader  Training  Applicable  Definitions 


Steps  in  Problem  Solving  *  A  logical,  systematic  process 
aimed  at  understanding  a  problem  issue  and  making  an 
informed  data  based  decision  to  solve  the  problem  or 
optimize  the  opportunity.  It  includes  six  steps  that  are: 
defining  the  problem,  analyzing  the  problem,  evaluating 
solutions,  planning  the  implementation,  and  implementing  the 
solution. 

Brainstorming  -  Verbal  generation  of  ideas  in  a  group  that 
key  the  identification  of  specific  details  of  a  topic 
through  encouraging  spontaneous,  rapid  and  creative  input. 
Judgement  is  deferred. 

Nominal  Group  Technique  (NGT)  -  A  method  of  reaching 
consensus  that  is  a  refinement  of  brainstorming.  It 
involves  the  individual  written  recording  of  ideas  followed 
by  a  group  discussion  that  ranks  and  prioritizes. 

Cause  and  Effect  Analysis  -  A  graphical  refinement  of 
brainstorming  designed  to  help  identify  the  root  causes  of  a 
problem.  Individuals  propose  causes  to  a  problem  that  are 
graphically  represented  as  branches  of  twigs  that  feed  into 
the  problem.  Also  known  as  fishbone  diagrams. 

Pareto  Analysis  -  A  prioritization  tool  involving  a 
graphical  technique  where  data  is  arranged  in  descending 
order  to  rank  the  contributors  to  a  problem  by  their  order 
of  importance. 

Checksheets  -  A  method  of  accumulating  and  documenting  data 
to  assist  in  the  detection  of  patterns  or  frequency  of 
occurrence  of  events. 

Flow  Charting  -  A  pictorial  representation  of  all  the  major 
sequential  steps  in  a  process. 

Solution  Selection  Matrix  -  A  numerically  comparative  chart 
that  rates  each  proposed  solution  with  respect  to  solution 
selection  criteria.  The  sum  of  rating  for  each  solution 
against  the  criteria  provides  a  structured  method  for 
comparison  of  solutions. 

Force  Field  Analysis  -  A  graphical  techniques  that 
identifies  the  driving  and  restraining  forces  that 
contribute  to  the  current  status  of  a  problem.  A  vertical 
line  represents  the  current  situation  and  drivers  and 
restrainers  are  opposed  to  each  other  on  opposite  sides  of 
the  line  to  represent  the  reason  for  the  current  status. 
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Action  Planning  -  A  written  plan  of  action  for  accomplishing 
a  stated  goal.  It  involves  specific  objectives,  required 
major  activities,  personnel  assignments  and 
responsibilities,  and  an  established  schedule  for 
accomplishment . 


r 
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Appendix  G:  CAT  Leader  Graduate  Questionnaire  Conaments 


Subject:  Overall  Comnents 

1.  The  training  program  provided  for  CAT  Leaders  is,  by 
textbook  example,  very  good.  However,  for  resolving 
most  of  the  types  of  problems  that  many  individuals  at 
the  organizational  level  are  asked  to  work,  the  CAT 
Training  Program  is  overkill. 

Recommend  that  ASD/TQ  conduct  a  survey  in  ASD, 
requesting  Program  Offices  to  identify  the  most  common 
types  of  problems  encountered.  That  is,  if  it  has  not 
yet  been  accomplished.  Then  restructure  training  for 
individuals  based  on  commonality  of  problems  to 
resolve. 

The  current  program  is  excellent  for  those  personnel 
that  are  directly  involved  in  the  re-organization/ 
coordination  of  the  AFLC  and  ASD  organization.  Those 
Structures/tasks/probl ems  are  very  complex. 

2.  Everything  okay. 

3.  This  (as  I  said  in  my  evaluation  of  the  course)  was  the 
best  course  on  this  subject  (problem  solving 
techniques)  that  I  have  ever  encountered.  I  believe 
(some  how)  we  should  equip  al 1  our  personnel  with  these 
tools . 

4.  I  have  my  first  CAT  involvement  within  the  next  week. 
Looking  forward  to  it.  It  was  hard  to  get  selected--no 
one  wants  to  allow  secretaries  activities  away  from 
their  office.  Not  even  those  working  for  their 
Master's  degree. 

5.  I  have  not  been  fully  utilized  as  a  CAT  Team  Leader, 
therefore  the  training  is  becoming  fuzzy.  I  feel  the 
money  spent  training  me  has  been  wasted  thus  far. 

6.  Future  courses  may  want  to  include  decision  making 
processes  where  consensus  is  not  a  requirement  or 
reaching  a  consensus  is  difficult. 

7.  Taking  the  course  material  and  making  it  work  in  your 
groups  requires  education  of  the  group.  Any  education 
materials  (plan)  you  could  provide  for  the  new  CAT 
leader  would  be  helpful.  Copying  the  CAT  manual  is  not 
appropriate,  but  I  have  made  selected  copies  of 
material  in  this  manual . 
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8. 


Excellent  training...  the  problem  is  applying  in  the 
organizational  environment!  Still  too  many  layers  of 
management  decision... 

9.  I  feel  that  the  participants  of  the  CAT  leader  course 
should  be  from  the  same  organization  rather  than 
several  different  ones.  That  way  people  get  to  know 
one  another  from  the  same  unit  and  work  on  a  final 
exercise  that  pertains  or  is  related  to  their 
organization. 

10.  I  feel  the  course  is  good  and  adequate. 

11.  Don't  have  any  specific  comments--other  than  I  have 
enjoyed  the  course  very  much--thought  it  was  very 
useful . 

12.  Need  more  guidance  on  how  to  deal  with  managements' 
resistance  when  teams  try  to  make  things  happen.  i.e. 
Education  showing  movies  monthly  on  TQ  working  in 
organizations  outside  the  government. 

13.  There  was  no  follow  up!  Need  references  and  other 
sources!  I  am  interested  in  the  outcome  of  this 
research. 

14.  TQ  seems  to  be  falling  from  popularity  already.  It  may 
only  be  in  the  organization  that  I'm  in. 

15.  The  initial  CAT  leader  training  or  a  refresher  course 
should  be  taken  immediately  prior  to  the  leading  of  a 
CAT  Team. 

Future  CAT  leaders  should  first  be  participant's  or 
first  hand  observers  in  on  going  CAT  teams  prior  to 
being  on  the  firing  line  as  a  CAT  leader. 

More  former  CAT  team  leaders  should  be  used  to  teach 
parts  of  the  leader  training  course. 

This  survey  should  be  done  incrementally:  1)  shortly 
after  the  course;  2}  after  CAT  team  participation;  3) 
after  CAT  team  leader  experience. 

16.  Our  office  has  practices  brainstorming  as  well  as  NGT 
on  our  off-site  meetings  but  nowhere  else.  The  mind 
set  is  such  that  these  valuable  techniques  be  only 
applied  to  TQ  related  activities,  not  training  others 
to  use  them  in  day-to-day  activities.  Promote  these 
techniques  via  training  all  the  unwashed  masses. 

17.  I  think  you  are  pretty  much  on  target  with  everything. 
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18.  Overall  a  good  course. 

19.  Need  more  specific  direction  in  conducting  a  CAT  and 
dealing  with  problems  encountered  during  a  CAT.  Also, 
how  it  interfaces  with  management. 

20.  The  training,  instructor,  class  materials  and 
environment  are  all  exceptional. 

The  biggest  obstacle  to  overcome  is  not  the  actual 
training,  which  is  actually  the  second  challenge,  but 
the  attitudes  and  "thinking**  of  our  current  work  force. 
Most  individuals  attending  the  training  are  people  from 
the  *'old**  school  of  thinking.  By  this  I  mean... gone  is 
the  day  when  a  person  can  achieve  a  job  or  task 
"alone**.  No  more  can  a  person  exist  in  a  world  of 
their  own,  in  a  little  office  doing  one  specialized 
mission.  Each  individual  must  be  diversified  and  have 
a  clear  understanding  of  many  processes  that  effect 
their  mission. 

Achievement  of  this  can  only  be  accomplished  by  working 
"together**.  Thus,  you  have  the  birth  of  not  only  TQ, 
but  a  whole  new  way  of  thinking.  When  people  are 
convinced  that  without  working  together  we  will  not 
only  "not  survive",  but  also  will  not  be  able  to 
achieve  "a  Total  Quality"  environment,  only  then  will 
they  accept  and  **use**  these  wonderful  tools  that  ASD 
has  endeavored  to  equip  us  with  to  achieve  this  goal. 

As  I  have  told  many  people,  [organization] * s  chief  was 
"TQ"  before  it  became  a  household  word  in  ASD.  But,  he 
perceived  his  mission  was  to  "convince"  his  people 
there  was  no  other  way  to  achieve  our  mission  without 
working  together,  which  in  turn  inspired  us  to  strive 
for  excellence  through  teamwork!  We  have  a  group  of 
people  with  good  "attitudes"  and  a  new  way  of  thinking. 
They  are  progressive  thinkers,  and  are  not  afraid  to 
try  new  methods.  Many  of  our  people  attended  your  CAT 
training.  They  came  back  truly  inspired.  I  felt  part 
of  it  because  they  saw  that  our  way  of  thinking  wan  now 
becoming  the  norm.  ASD  was  now  on  the  road  to 
providing  the  tools  for  all  people  to  have  this  "new 
thinking .  ** 

Once  people  accept  the  idea  as  a  "way  of  life",  they 
will  use  the  wonderful  "tools"  that  ASD  is  providing. 

21.  Some  sort  of  follow-up  training  should  be  accomplished. 
I  have  volunteered  myself  as  a  CAT  leader  member  and 
never  had  the  opportunity  of  using  any  of  the  skills  I 
have  learned  at  the  CAT  course. 


157 


22.  I  haven't  been  involved  in  a  CAT  yet,  so  much  of  my 
answers  were  either  undecided  or  could  have  been 
answered  better  if  I  was  more  involved.  Seems  like  the 
same  people  do  the  CATs  all  the  time. 

23.  Need  something  on  difficult  people.  Management  needs 
to  be  educated--impl ementation  of  recommendations  is 
their  department  not  the  CAT  teams.  "Don't  kill  the 
messenger  because  you  don't  like  the  findings." 

24.  [organization]  middle  management  not  sold/not  using. 
Upper  level? 

25.  It  made  me  a  believer  in  using  intervals!  Way  to  go! 

26.  All  sections  very  good.  They  reinforced  my  belief  that 
the  biggest  roadblock  to  getting  the  job  done  is 
interpersonal  relations,  not  technical  inadequacies. 

27.  CAT  leader  training  has  proven  to  be  the  most  useful 
training  I  have  had  at  ASD.  Strongly  recommend  this 
training  for  everyone . 

28.  I  thought  the  course  and  lessons  were  fine  but  haven't 
had  the  opportunity  to  employ  them.  I  believe  TQ  at 
ASD  has  taken  a  serious  setback  in  the  last  8-9  months. 

29.  Have  not  been  selected  to  support  CAT  effort  and  would 
require  a  review  on  all  areas  before  I  could  be  a 
significant  team  member.  I  endorse  the  CAT  initiative 
but  feel  that  many  people  are  afraid  to  make  a 
decision.  Also  maybe  the  teams  are  too  large  to  be 
effective 

30.  1)  [more  info  on]  how  to  recruit  CAT  team  members;  2)  a 
little  more  emphasis  should  be  made  regarding  "how  to 
wrap-up  a  CAT"  besides  recommendations  and 
presentations;  3)  [more  info  on]  how  to  get  support 
from  2-letter. 

31.  Enjoyed  it! 

32.  All  sections  excellent.  TQ  problem  solving  not  big  in 
[org].  TQ  education  is  good  in  [org]. 

33.  How  about  a  "fun"  test  or  exam  so  we  can  be  challenged 
to  be  kept  current.  Could  be  on  AMS  and  you  could 
monitor  trends  by  answers  received.  Sponsor  a  Wright- 
Patt  wide  CAT  leader  alumni  association  and  have 
someone  record  lessons  learned  and  shared. 
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34.  CAT  leader  training  should  be  3  days  and  not  four  days 
long . 

35.  It  took  me  40  minutes  to  complete  this  survey  with  one 
2-minute  interruption. 

36.  Give  a  framework  for  starting  teams.  Trainers  just 
presented  tools,  taught  how  to  use  them  and  said  go 
"build  a  house."  When  we  asked  "where  do  we  start?" 
they  said,  "nothing  is  right  or  wrong  just  do  what 
feels  right."  Suggestions  of  minimum  start-up  tasks 
would  be  good. 

37.  I  see  no  reason  to  change  anything! 

38.  Overall,  I  believe  the  CAT  leader  training  was  very 
useful  training  for  ASD  personnel.  The  lessons  on 
problem  solving  techniques  were  very  useful. 

Widely  providing  training  on  methods  for  fixing 
processes  suggests  there  might  be  a  correspondingly 
wide  opportunities  for  fixing  processes... 

Many  processes  we  use  are  dictated  externally  through 
regulation,  law,  etc.  I  know  of  no  significant 
processes  or  problems  which  have  been  changed  as  a 
result  of  CAT  leader  training  or  TQ. . . 

[I  tried  group  consensus  on  my  project  team  with  poor 
results]...  Team  consensus  works  if  the  team  is  only 
composed  of  reasonable  people,  with  like  viewpoints, 
who  are  willing  to  compromise.  That  would  be  a  rare 
and  probably  unimaginative  team.  I  have  concluded  that 
team  leaders  need  to  try  to  reach  consensus,  but  must 
be  allowed  and  willing  to  make  decisions  after  careful 
consideration  of  team  viewpoints.  Undoing  consensus 
approach,  once  started  is  difficult.  I  recommend  CAT 
leader  training  be  very  careful  about  encouraging 
consensus  building  as  a  sole  method  of  problem  solving 
to  the  exclusion  of  decision  making. 

39.  Overall,  it  was  an  excellent  training  opportunity. 

40.  Good  coverage  (both  time  and  material)  of  all  sections. 

41.  Necessary  to  take  the  time  to  use  skills  learned,  its 
common  to  have  the  desire  to  use  techniques  but 
difficult  to  rearrange  busy  schedules  to  implement. 

42.  Need  to  send  this  survey  out  sooner,  say  less  than  two 
weeks  after  completion.  My  memory  is  somewhat  vague  on 
making  recommendations  for  improvements  to  TQ  class. 
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43. 


"Enthusiasm 


Keep  enthusiastic  teachers  for  the  course, 
is  contagious." 

44.  I  found  the  overall  training  valuable  outside  the 
context  of  a  specific  CAT  situation.  (even  useful  in 
my  community  volunteer  work!)  Thanks. 

45.  Overall  Super  job 

46.  Generally  useful 

47.  Class  too  long  (4  days)  Bring  more  real  (ASD)  issues 
into  discussion. 

48.  My  biggest  problem  is  I  was  a  CAT  leader  before 
training  and  now  I  haven't  been  given  a  CAT  recently. 
The  training  needs  to  be  given  right  before  the  team  is 
commissioned . 

49.  Can  this  course  be  distilled/shortened  at  all?  I 
experienced  a  tough  time  getting  four  days  off  for 
training  from  my  boss,  TQ  or  not. 

50.  I  have  used  many  of  the  things  I  learned  in  CAT 
training  in  my  work  and  in  my  personal  life.  I 
recommend  it  for  everyone  whether  they  are  TQ  related 
or  not. 

51.  Not  every  SFO  or  problem  requires  the  long,  tedious  CAT 
process  to  apply  an  effective  solution.  In  most  cases 
--  a  single  action  officer  can  solve  and  implement  a 
very  effective  fix  to  a  process  problem.  The  democracy 
fails  at  times.  Socialism  is  often  abused  by 
individuals  seeking  power  and  attention.  The  CAT 
process  takes  an  inordinate  amount  of  time--  much  too 
tedious  and  long. 

52.  I  thought  the  CAT  leader  training  was  excellent  and 
should  be  made  mandatory  for  all  EN  branch  chiefs. 
Everything  we  do  at  ASD  involves  working  with  teams  of 
people.  Therefore,  the  CAT  leader  training  should  be 
given  throughout  ASD/EN. 

Subject:  TQ  Background  Section 

1.  Course  I  went  to  -  discussion  centered  on  Japanese 
quality  initiatives  and  on.  Feedback  I  received  from 
others  who  had  same  training,  but  different  instructors 
was  that  too  much  time  was  spent  covering  the 
fundamentals  AGAIN! 

2.  Should  be  shorter--an  overview. 


160 


3.  Old  hat  for  many,  but  must  be  reviewed  for  folks  new  to 
TQ. 

4.  Too  lengthy. 

5.  Tailor  to  the  group. 

€.  Explain  more  about  the  Cumberland  Group,  where  they 
started  and  how  we  got  'em. 

7.  Most  important  for  SPO  wide  CAT  training. 

8.  Improve  presentation,  book  was  not  organized  logically 
when  I  took  the  course. 

9.  Should  be  separate  from  CAT  leader  training. 

10.  Give  us  some  examples  of  some  successful  CATs  and  CATs 
in  progress  for  motivation. 

11.  Terrible,  need  to  get  with  TQ  office  and  get  up  to 
date . 

12.  People  with  similar  levels  of  TQ  exposure  should  be 
grouped  together  for  classes  and  this  topic  structured 
accordingly . 

13.  Better  definition  of  the  process,  who  are  the  points 
of  contact. 

14.  Should  have  more  (all  employees  need  a  40  hour  course 
explaining  the  TQ  process). 

15.  Too  narrow  a  view.  The  Cumberland  Group  approach  is  a 
misapplication  of  factory  TQM  to  white  collar  TQM 
organizations.  A  broader  overview  with  specific 
references  to  Deming,  Crosby,  etc.  would  be  more 
valuable. 

16.  Do  not  assume  everyone  is  completely  familiar  with  TQ. 

17.  More  on  basics  of  TQ.  I  was  sent  to  CAT  training  with 
zero  info  on  what  TQ  was  functionally. 

18.  Needed  to  go  into  more  depth  and  not  assume  people  were 
all  knowledgeable  on  TQ. 

Subject:  Personal  Values 

1.  Excellent.  A  root  cause  of  many  team  problems  and 

disagreements.  Not  fully  appreciated  or  used  by  most 
trainees.  The  personality  tests  red-green-blue  was  not 
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very  helpful--  Myers-Briggs  would  be  better,  more 
usable,  and  widely  recognized. 

2.  Interesting.  Good. 

3.  Good  input. 

4.  Informative. 

5.  Showing  how  values  related  to  lifetime  goals  was 
important.  Then,  I  was  better  able  to  understand  the 
goals  setting  process  in  the  work  place. 

6.  Excellent--  sets  the  stage. 

7.  Good  section  to  help  people  better  understand  how  the 
way  people  are  raised  affects  how  they  will  work 
together  on  a  team. 

8.  Inappropriate  and  unprofessional  examples  used  to  make 
a  point.  Why  "not  have  sex  with  others" 

9.  Important  to  learn  about  oneself. 

10.  Very  valuable 

11.  No  preaching 

12.  Terrific--  opened  my  eyes. 

Subject:  Group  Dynamics 

1.  Needed  more  time. 

2.  OK,  but  could  be  much  improved  by  reviewing  current 
research  and  books  on  high  performance  teams.  The 
focus  on  the  stages  the  teams  go  thru  is  not  enough. 

3.  How  about  role  playing  to  deal  with  difficult  team 
members? 

4.  This  was  especially  interesting  to  me,  because  it 
clarified  to  me  how  I  come  across  to  others 
(clarifying/dominating) . 

5.  Sometimes  kind  of  technical. 

6.  Vital. 

7.  Terrific--  opened  my  eyes. 
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Subject:  Steps  of  Problem  Solving 

1.  Needed  more  time. 

2.  Condensed  handout  for  group  distribution. 

3.  Some  seem  more  useful  than  others. 

4.  Extremely  important.  Able  to  use  if  able  to  remember 
this  process.  Suggest  a  summary  card  be  printed  and 
handed  out  to  students.  Could  post  card  on  desk. 

5.  Vital. 

6.  Now  I  know  how  to  approach  problem  solving  in  an 
orderly  and  logical  way. 

Subject:  Tools  and  Techniques 

1.  Good  coverage. 

2.  Condensed  handout  for  group  distribution  and  education 
of  team. 

3.  OK.  Need  more  examples  and  explanation  of  when  to  use. 
Why  should  you  choose  one  method  over  another. 

4.  The  fishbone  diagrams,  pareto  analysis,  solution 
selection  matrix,  and  force  field  analysis  are  nice 
concepts  but  do  not  work  like  the  human  mind  so  they 
seem  like  more  work  than  they  are  worth.  Nobody  ever 
uses  them. 

5.  More  background  info  would  be  useful.  I  knew  what  was 
going  on  because  of  previous  training,  but  others  were 
1  ost . 

6.  Need  to  weed  out  least  used  ones  and  spend  more  time  on 
others . 

7.  Skinny  down,  clean  up  manual. 

8.  Could  have  been  explained  better.  In  some  cases,  over¬ 
simplification  complicated  explanation. 

9.  If  you  try  to  use  any  of  these  techniques  in  a  non-TQ 
meeting,  people  think  you  are  trying  to  "force  TQ"  upon 
them.  Suggest  application  methods  in  meeting 
situations . 

10.  One  of  the  most  helpful. 
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11.  Emphasize  this  more.  As  new  techniques  are  learned, 
make  sure  all  CAT  leaders  are  educated. 

12.  Don’t  overload  people  all  at  once. 

13.  Need  better  examples  or  explanations--  especially 
Pareto  Analysis. 

14.  Vital  (could  use  a  summary  card  for  quick  reference  at 
the  job  site. 

15.  Copier  problem  overworked. 

16.  So-so. 


Subject:  Exercises 


1.  Not  all  related  to  our  environment. 

2.  OK,  but  rushed.  Instructor  with  our  group  didn't  seem 
interested  and  wasn't  helpful. 

3.  Very  effective  and  enjoyable.  Kept  my  attention. 

4.  Would  rather  work  on  [examples]  more  relevant  to  the 
work  place. 

5.  Desert  Storm  exercise  was  inappropriate  example  for 
synergy.  Scoring  system  was  flawed. 

6.  Good  for  working  through  the  process  without  a  lot  of 
stress . 


7.  Very  effective. 

8.  Some  good,  some  very  flimsy  data  to  begin  with. 


9. 


This  above  all,  probably  gives  greatest  applicability 
back  to  the  work  environment.  However  the  exercises 
need  to  be  related  to  work  (applicable)  and  substantial 
(not  fluff). 


Subject:  Movies 


1.  Good  examples. 


2.  There  is  a  definite  lack  of  QUALITY  TQ  movies  available 
for  general  use.  The  movies  in  the  CAT  course  weren't 
too  bad. 


3.  A  waste  of  time. 
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4.  Paradigm  film  was  outstanding. 

5.  All  movies  should  be  shown  in  one  type  of  training 
course  or  another--  too  much  duplication. 

6.  Don't  remember  seeing  any  movies.  Would  be  a  good  tool 
(ex:  CAT  team  in  action  solving  a  problem). 

7.  Meeting  Robbers  is  great. 

8.  The  Abilene  Paradox--  great. 

Subject:  Final  Exercise 

1.  Good  approach. 

2.  This  exercise  was  great  in  that  it  tied  all  the 
information  into  practice.  A  true  synergy  example. 

3.  Our  group  presentation  was  a  disaster  because  of  one 
individual.  I  don't  know  what  you  can  do  to  correct 
those  types  of  problems. 

4.  Took  my  group  all  day  and  still  had  problems.  In  the 
final  problem,  the  number  of  customers  at  the  base 
cafeteria  was  declining.  I  had  difficulty  grasping  why 
this  was  a  problem.  If  one  base  cafeteria  is  no  longer 
needed,  then  that's  a  savings  to  the  U.S.  taxpayers. 
Others  in  the  group  didn't  seem  to  have  any  difficulty 
in  seeing  the  problem,  perhaps  because  we  were  required 
to  have  a  solution  in  a  few  hours.  Something  needs  to 
be  done  to  get  a  clear  concept  of  the  mission.  At  one 
point  in  the  class,  visiting  CAT  team  leaders  were 
asked  to  give  their  experiences  as  a  CAT  leader  using 
TQM.  One  individual  was  quite  proud  of  her 
accomplishment  of  pushing  routine  paperwork  away  from  a 
secretary  making  $10/hr  onto  a  manager  making  $50/hi.  . 

As  it  turned  out,  the  occurrence  of  declining  customers 
at  the  cafeteria  was  due  to  a  safety  problem.  Up 
front,  the  goal  of  the  group  should  have  been  to 
discover  in  any  health  or  safety  dangers  existed  at  the 
cafeteria  and  not  to  increase  clientele. 

5.  Need  a  more  relevant  problem  than  bad  coffee. 

6.  Overhearing  other  groups’  progress  ruins  the 
effectiveness  of  the  exercise. 

7.  Excellent!  Could  apply  everything  that  was  taught-- 
brought  everything  together. 

8.  Would  rather  work  one  more  relevant  to  work  place. 
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9. 


This  exercise  was  good.  However,  I  would  schedule  it 
around  lunch.  The  final  day  should  include  a  time  when 
cohesiveness  of  class  members  should  material ize 
without  stress.  A  working  lunch  is  very  stressful  to 
me.  People  need  a  break  during  the  day  to  renew 
themselves  so  that  they  can  operate  at  peak  performance 
(TQ)  during  the  afternoon  as  well  as  during  the 
morning . 

10.  Interesting. 

11.  Well  done  and  vital  to  get  the  hand-on  experience  that 
fixes  the  concepts  in  your  mind. 

12.  Needs  work. 

13.  Certainly  brings  group  dynamics  into  use. 

14.  This  above  all,  probably  gives  greatest  applicability 
back  to  the  work  environment.  However  the  exercises 
need  to  be  related  to  work  (applicable)  and  substantial 
(not  fluff) . 

15.  Had  to  wait  on  other  groups  to  see  the  leader 
(facilitator)  which  impacted  time  spent  on  exercise. 


( 
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